[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:03:38 dist-test-slave-hlq1 krb5kdc[1786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:03:38 dist-test-slave-hlq1 krb5kdc[1786](info): set up 2 sockets
Jun 21 07:03:38 dist-test-slave-hlq1 krb5kdc[1786](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 07:03:39.869822 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.007s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:03:40 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:03:40.084301 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41217
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:40.466811 1799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:40.467166 1799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:40.467268 1799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:40.475526 1799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:03:40.475672 1799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:40.475724 1799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:40.475780 1799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:03:40.475828 1799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:03:40.486508 1799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:40.488240 1799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:40.489890 1799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:40.503574 1807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:40.503650 1805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:40.503855 1804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:40.504532 1799 server_base.cc:1034] running on GCE node
I20260621 07:03:40.505656 1799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:40.507576 1799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:40.508787 1799 hybrid_clock.cc:648] HybridClock initialized: now 1782025420508712 us; error 71 us; skew 500 ppm
I20260621 07:03:40.509284 1799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:40 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025420, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:40.513418 1799 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:03:40.515228 1799 webserver.cc:460] Webserver started at http://127.1.188.254:44565/ using document root <none> and password file <none>
I20260621 07:03:40.516172 1799 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:40.516315 1799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:40.516693 1799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:40.520355 1799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "842453e62ec34c1eb6f353b90471e72c"
format_stamp: "Formatted at 2026-06-21 07:03:40 on dist-test-slave-hlq1"
server_key: "75777713147e34cedd24b3f84bb5ce03"
server_key_iv: "88e1035aeb2c2ef849dbafbdeda5637d"
server_key_version: "encryptionkey@0"
I20260621 07:03:40.521220 1799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "842453e62ec34c1eb6f353b90471e72c"
format_stamp: "Formatted at 2026-06-21 07:03:40 on dist-test-slave-hlq1"
server_key: "75777713147e34cedd24b3f84bb5ce03"
server_key_iv: "88e1035aeb2c2ef849dbafbdeda5637d"
server_key_version: "encryptionkey@0"
I20260621 07:03:40.527565 1799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260621 07:03:40.531770 1814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:40.533533 1799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:03:40.533767 1799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "842453e62ec34c1eb6f353b90471e72c"
format_stamp: "Formatted at 2026-06-21 07:03:40 on dist-test-slave-hlq1"
server_key: "75777713147e34cedd24b3f84bb5ce03"
server_key_iv: "88e1035aeb2c2ef849dbafbdeda5637d"
server_key_version: "encryptionkey@0"
I20260621 07:03:40.534024 1799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:40.585254 1799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:40.590775 1799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:40.591140 1799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:40.614017 1799 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41217
I20260621 07:03:40.614037 1865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41217 every 8 connection(s)
I20260621 07:03:40.615968 1799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:03:40.617630 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 1799
I20260621 07:03:40.618005 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:03:40.620889 1779 external_mini_cluster.cc:1428] Setting key 5f5d5d393e541ee4f70e99d2619fe429
I20260621 07:03:40.622141 1866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:40.634972 1866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c: Bootstrap starting.
I20260621 07:03:40.639600 1866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:40.641144 1866 log.cc:824] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:40.644621 1866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c: No bootstrap required, opened a new log
I20260621 07:03:40.653265 1866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } }
I20260621 07:03:40.653748 1866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:40.653864 1866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842453e62ec34c1eb6f353b90471e72c, State: Initialized, Role: FOLLOWER
Jun 21 07:03:40 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:40.654807 1866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } }
I20260621 07:03:40.655133 1866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:03:40.655282 1866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:03:40.655501 1866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:40.658061 1866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } }
I20260621 07:03:40.658725 1866 leader_election.cc:302] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842453e62ec34c1eb6f353b90471e72c; no voters:
I20260621 07:03:40.659530 1866 leader_election.cc:288] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:03:40.659636 1871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:40.660948 1871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [term 1 LEADER]: Becoming Leader. State: Replica: 842453e62ec34c1eb6f353b90471e72c, State: Running, Role: LEADER
I20260621 07:03:40.661657 1871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } }
I20260621 07:03:40.662284 1866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:03:40.664868 1873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842453e62ec34c1eb6f353b90471e72c. Latest consensus state: current_term: 1 leader_uuid: "842453e62ec34c1eb6f353b90471e72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } } }
I20260621 07:03:40.665287 1873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:40.668269 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:40.626574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.1.188.254:41217)
0621 07:03:40.627725 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:40.627745 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:40.629286 (+ 1541us) server_negotiation.cc:407] Connection header received
0621 07:03:40.639562 (+ 10276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:40.639602 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:40.640235 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:40.640760 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:40.643934 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:40.646049 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:40.647883 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:40.649024 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:40.649212 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:40.655982 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:40.655992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:40.656007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:40.656048 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:40.660103 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:40.661345 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:40.661354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:40.661364 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:40.661625 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:40.662216 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:40.662221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:40.662224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:40.663015 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:40.663348 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:40.665585 (+ 2237us) server_negotiation.cc:299] Negotiation successful
0621 07:03:40.666206 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":528,"threads_started":1}
I20260621 07:03:40.669587 1872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842453e62ec34c1eb6f353b90471e72c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842453e62ec34c1eb6f353b90471e72c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41217 } } }
I20260621 07:03:40.669947 1872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:40.680059 1876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:03:40.688735 1876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:03:40.700390 1876 catalog_manager.cc:1347] Generated new cluster ID: 48e539ca18c94fa29dfd5d62bc375d73
I20260621 07:03:40.700553 1876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:03:40.720182 1876 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:03:40.721725 1876 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:03:40.739737 1876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842453e62ec34c1eb6f353b90471e72c: Generated new TSK 0
I20260621 07:03:40.740940 1876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:03:40.804816 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:41.218662 1893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:41.219045 1893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:41.219237 1893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:41.226918 1893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:41.227072 1893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:41.227250 1893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:03:41.237478 1893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:41.239058 1893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:41.240653 1893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:41.254467 1901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:41.254585 1899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:41.255064 1898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:41.255220 1893 server_base.cc:1034] running on GCE node
I20260621 07:03:41.255911 1893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:41.256958 1893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:41.258244 1893 hybrid_clock.cc:648] HybridClock initialized: now 1782025421258201 us; error 35 us; skew 500 ppm
I20260621 07:03:41.258594 1893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:41 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:41.266004 1893 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:03:41.267812 1893 webserver.cc:460] Webserver started at http://127.1.188.193:36711/ using document root <none> and password file <none>
I20260621 07:03:41.268581 1893 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:41.268739 1893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:41.269068 1893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:41.272217 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "824a9ecfebd64950a261e2bdbf97066c"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "dc506cd127028825861a28e1fd07a789"
server_key_iv: "0a3c3b3df2c42380ce9932228cd59809"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.272972 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "824a9ecfebd64950a261e2bdbf97066c"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "dc506cd127028825861a28e1fd07a789"
server_key_iv: "0a3c3b3df2c42380ce9932228cd59809"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.278513 1893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:03:41.282140 1908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.283663 1893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:03:41.283840 1893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "824a9ecfebd64950a261e2bdbf97066c"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "dc506cd127028825861a28e1fd07a789"
server_key_iv: "0a3c3b3df2c42380ce9932228cd59809"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.284056 1893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:41.304841 1893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:41.309438 1893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:41.309754 1893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:41.310786 1893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:41.312346 1893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:41.312461 1893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.312615 1893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:41.312693 1893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.347504 1893 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:39247
I20260621 07:03:41.347539 2020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:39247 every 8 connection(s)
I20260621 07:03:41.348884 1893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:03:41.358389 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 1893
I20260621 07:03:41.358644 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:03:41.359284 1779 external_mini_cluster.cc:1428] Setting key f67a46fb0d28a20fac3002cbd72d8da3
Jun 21 07:03:41 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:41.369655 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:41.351576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:55497 (local address 127.1.188.254:41217)
0621 07:03:41.352076 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:41.352081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:41.352978 (+ 897us) server_negotiation.cc:407] Connection header received
0621 07:03:41.354893 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:41.354899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:41.355009 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:41.355149 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:41.356746 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.357490 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.359224 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.359707 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.359848 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:41.363564 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:41.363575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:41.363578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:41.363612 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:41.366448 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:41.367218 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:41.367226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:41.367234 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:41.367355 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:41.367871 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:41.367878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:41.367885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:41.368167 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:41.368325 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:41.369043 (+ 718us) server_negotiation.cc:299] Negotiation successful
0621 07:03:41.369286 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260621 07:03:41.370680 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:41.352058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41217 (local address 127.1.188.193:55497)
0621 07:03:41.352806 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:41.352847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:41.354589 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:41.355386 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:41.355446 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:41.355984 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:41.356557 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:41.356574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.357673 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.357678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:41.359071 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:41.359083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.359940 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.359949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:41.360197 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:41.360900 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:41.360929 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:41.363311 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:41.366628 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:41.366637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:41.366655 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:41.367082 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:41.367538 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:41.367543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:41.367547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:41.367740 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:41.368296 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:41.368304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:41.368662 (+ 358us) client_negotiation.cc:769] Sending connection context
0621 07:03:41.368936 (+ 274us) client_negotiation.cc:240] Negotiation successful
0621 07:03:41.369460 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":274,"threads_started":1}
I20260621 07:03:41.372432 2021 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41217
I20260621 07:03:41.372777 2021 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:41.373539 2021 heartbeater.cc:511] Master 127.1.188.254:41217 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:03:41.376267 1831 ts_manager.cc:194] Registered new tserver with Master: 824a9ecfebd64950a261e2bdbf97066c (127.1.188.193:39247)
I20260621 07:03:41.377961 1831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:55497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:03:41.399295 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:41.752727 2027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:41.753028 2027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:41.753170 2027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:41.760968 2027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:41.761088 2027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:41.761216 2027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:03:41.771576 2027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:41.772984 2027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:41.774492 2027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:41.788138 2032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:41.788129 2033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:41.788944 2035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:41.789351 2027 server_base.cc:1034] running on GCE node
I20260621 07:03:41.789983 2027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:41.791002 2027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:41.792279 2027 hybrid_clock.cc:648] HybridClock initialized: now 1782025421792209 us; error 97 us; skew 500 ppm
I20260621 07:03:41.792666 2027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:41 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:41.795925 2027 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:03:41.797595 2027 webserver.cc:460] Webserver started at http://127.1.188.194:33413/ using document root <none> and password file <none>
I20260621 07:03:41.798293 2027 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:41.798400 2027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:41.798727 2027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:41.801957 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "3431f2d40dd84812b0b7d84b6b10f902"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "2dbb0f11dbc65450f5b53b3246943563"
server_key_iv: "a90ba740d1d1325dd8df2ba663086f4d"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.802858 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "3431f2d40dd84812b0b7d84b6b10f902"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "2dbb0f11dbc65450f5b53b3246943563"
server_key_iv: "a90ba740d1d1325dd8df2ba663086f4d"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.808763 2027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 07:03:41.812810 2042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.814217 2027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:03:41.814409 2027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "3431f2d40dd84812b0b7d84b6b10f902"
format_stamp: "Formatted at 2026-06-21 07:03:41 on dist-test-slave-hlq1"
server_key: "2dbb0f11dbc65450f5b53b3246943563"
server_key_iv: "a90ba740d1d1325dd8df2ba663086f4d"
server_key_version: "encryptionkey@0"
I20260621 07:03:41.814602 2027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:41.825887 2027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:41.830133 2027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:41.830438 2027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:41.831475 2027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:41.832775 2027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:41.832847 2027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.832973 2027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:41.833024 2027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:41.867038 2027 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:46877
I20260621 07:03:41.867084 2154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:46877 every 8 connection(s)
I20260621 07:03:41.868384 2027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:03:41.869127 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2027
I20260621 07:03:41.869417 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:03:41.869971 1779 external_mini_cluster.cc:1428] Setting key 0791253bf1ec7e7adf9f11186cbe1f49
Jun 21 07:03:41 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025421, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:03:41.894567 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:41.871347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:39035 (local address 127.1.188.254:41217)
0621 07:03:41.871860 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:41.871867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:41.873118 (+ 1251us) server_negotiation.cc:407] Connection header received
0621 07:03:41.876205 (+ 3087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:41.876209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:41.876293 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:41.876393 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:41.878396 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.879255 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.881939 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.882418 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.882528 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:41.887244 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:41.887249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:41.887252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:41.887299 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:41.890471 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:41.891587 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:41.891594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:41.891597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:41.891771 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:41.892293 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:41.892302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:41.892308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:41.892720 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:41.892897 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:41.893745 (+ 848us) server_negotiation.cc:299] Negotiation successful
0621 07:03:41.894030 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":441,"threads_started":1}
I20260621 07:03:41.895624 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:41.872038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41217 (local address 127.1.188.194:39035)
0621 07:03:41.872939 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:41.872992 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:41.875856 (+ 2864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:41.876689 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:41.876704 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:41.877477 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:41.878202 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:41.878222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.879456 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.879464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:41.881757 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:41.881776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:41.882679 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:41.882688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:41.883042 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:41.883872 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:41.883907 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:41.886957 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:41.890645 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:41.890653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:41.890668 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:41.891385 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:41.891934 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:41.891938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:41.891940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:41.892153 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:41.892885 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:41.892891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:41.893317 (+ 426us) client_negotiation.cc:769] Sending connection context
0621 07:03:41.893589 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 07:03:41.894030 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":351,"threads_started":1}
I20260621 07:03:41.897392 2155 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41217
I20260621 07:03:41.897853 2155 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:41.898663 2155 heartbeater.cc:511] Master 127.1.188.254:41217 requested a full tablet report, sending...
I20260621 07:03:41.900190 1831 ts_manager.cc:194] Registered new tserver with Master: 3431f2d40dd84812b0b7d84b6b10f902 (127.1.188.194:46877)
I20260621 07:03:41.901026 1831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:39035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:03:41.914337 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:42.228645 2161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:42.228905 2161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:42.229059 2161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:42.236263 2161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:42.236380 2161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:42.236549 2161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:03:42.246505 2161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41217
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:42.247941 2161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:42.249294 2161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:42.262295 2166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:42.262382 2169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:42.262270 2167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:42.263255 2161 server_base.cc:1034] running on GCE node
I20260621 07:03:42.263934 2161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:42.264801 2161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:42.266028 2161 hybrid_clock.cc:648] HybridClock initialized: now 1782025422265978 us; error 48 us; skew 500 ppm
I20260621 07:03:42.266361 2161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:42.269876 2161 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:03:42.271443 2161 webserver.cc:460] Webserver started at http://127.1.188.195:37627/ using document root <none> and password file <none>
I20260621 07:03:42.272125 2161 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:42.272267 2161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:42.272574 2161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:42.275497 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "064830f347eb44248c08a22cbf6e186e"
format_stamp: "Formatted at 2026-06-21 07:03:42 on dist-test-slave-hlq1"
server_key: "39e6c2e63436129596cc555c529d7656"
server_key_iv: "c969421fd0faaa9c5170fa2f9e78a6f9"
server_key_version: "encryptionkey@0"
I20260621 07:03:42.276294 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "064830f347eb44248c08a22cbf6e186e"
format_stamp: "Formatted at 2026-06-21 07:03:42 on dist-test-slave-hlq1"
server_key: "39e6c2e63436129596cc555c529d7656"
server_key_iv: "c969421fd0faaa9c5170fa2f9e78a6f9"
server_key_version: "encryptionkey@0"
I20260621 07:03:42.281661 2161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:03:42.285195 2176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:42.286502 2161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:03:42.286700 2161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "064830f347eb44248c08a22cbf6e186e"
format_stamp: "Formatted at 2026-06-21 07:03:42 on dist-test-slave-hlq1"
server_key: "39e6c2e63436129596cc555c529d7656"
server_key_iv: "c969421fd0faaa9c5170fa2f9e78a6f9"
server_key_version: "encryptionkey@0"
I20260621 07:03:42.286893 2161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:42.307605 2161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:42.311683 2161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:42.312027 2161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:42.312997 2161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:42.314335 2161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:42.314443 2161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:42.314586 2161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:42.314684 2161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:42.348217 2161 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:38891
I20260621 07:03:42.348217 2288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:38891 every 8 connection(s)
I20260621 07:03:42.349886 2161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:03:42.350723 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2161
I20260621 07:03:42.350919 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:03:42.351505 1779 external_mini_cluster.cc:1428] Setting key 13cce8cc1e1c38bfbce67f7678b75c7c
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:42.372875 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.354155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:50223 (local address 127.1.188.254:41217)
0621 07:03:42.354356 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.354363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.356991 (+ 2628us) server_negotiation.cc:407] Connection header received
0621 07:03:42.358797 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.358802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.358879 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.358976 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.360513 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.361286 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.362867 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.363388 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.363557 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.367111 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.367116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.367119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.367186 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.370218 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.370866 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.370869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.370872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.370985 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.371445 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.371453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.371459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.371738 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.371863 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.372467 (+ 604us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.372629 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:03:42.373857 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.353554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41217 (local address 127.1.188.195:50223)
0621 07:03:42.356827 (+ 3273us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:42.356871 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:42.358538 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:42.359184 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:42.359192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:42.359784 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:42.360335 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:42.360352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.361414 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.361417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:42.362743 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:42.362754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.363696 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.363706 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:42.363919 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.364608 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:42.364641 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:42.366910 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:42.370340 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:42.370347 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:42.370360 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:42.370772 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:42.371070 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:42.371073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:42.371075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:42.371303 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:42.371845 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:42.371851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:42.372149 (+ 298us) client_negotiation.cc:769] Sending connection context
0621 07:03:42.372374 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 07:03:42.372732 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2983,"thread_start_us":2742,"threads_started":1}
I20260621 07:03:42.375322 2289 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41217
I20260621 07:03:42.375645 2289 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:42.376381 2289 heartbeater.cc:511] Master 127.1.188.254:41217 requested a full tablet report, sending...
I20260621 07:03:42.377844 1831 ts_manager.cc:194] Registered new tserver with Master: 064830f347eb44248c08a22cbf6e186e (127.1.188.195:38891)
I20260621 07:03:42.378687 1831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:50223
I20260621 07:03:42.381065 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:03:42.381374 2021 heartbeater.cc:503] Master 127.1.188.254:41217 was elected leader, sending a full tablet report...
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:42.425766 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.410834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.1.188.193:39247)
0621 07:03:42.411558 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.411565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.411594 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:03:42.411778 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.411785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.412002 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.412187 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.413570 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.414576 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.415891 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.416453 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.416555 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.420481 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.420491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.420504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.420536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.423315 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.423959 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.423966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.423968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.424085 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.424412 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.424415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.424417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.424726 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.424958 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.425183 (+ 225us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.425375 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":343,"threads_started":1}
W20260621 07:03:42.427042 1936 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35746
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:42.471494 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.456866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.1.188.193:39247)
0621 07:03:42.457093 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.457101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.457332 (+ 231us) server_negotiation.cc:407] Connection header received
0621 07:03:42.457552 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.457561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.457661 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.457778 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.459305 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.460109 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.461365 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.461778 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.461915 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.466058 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.466063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.466067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.466105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.468689 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.469457 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.469465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.469469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.469603 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.469978 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.469984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.469987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.470294 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.470549 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.470848 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.471081 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:03:42.473145 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 1893
I20260621 07:03:42.486593 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.474216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.1.188.193:39247)
0621 07:03:42.474415 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.474420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.474634 (+ 214us) server_negotiation.cc:407] Connection header received
0621 07:03:42.474799 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.474810 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.474864 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.474961 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.476205 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.476910 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.478197 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.478621 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.478749 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.481340 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.481347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.481353 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.481390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.483946 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.484581 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.484588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.484592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.484763 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.485125 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.485130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.485134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.485436 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.485707 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.486010 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.486235 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:03:42.487603 2016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:42.571228 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 1893
I20260621 07:03:42.594025 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2027
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:03:42.610723 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.595084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.1.188.194:46877)
0621 07:03:42.595764 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.595771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.595798 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:03:42.595859 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.595865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.596078 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.596227 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.597470 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.598633 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.600037 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.600636 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.600751 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.604723 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.604735 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.604749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.604784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.608164 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.608807 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.608814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.608816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.608936 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.609293 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.609296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.609298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.609634 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.609863 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.610117 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.610303 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":390,"threads_started":1}
I20260621 07:03:42.612217 2150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:42.700273 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2027
I20260621 07:03:42.716939 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2161
Jun 21 07:03:42 dist-test-slave-hlq1 krb5kdc[1786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:03:42.734200 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:42.718052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.1.188.195:38891)
0621 07:03:42.718726 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:42.718734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:42.718764 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:03:42.718823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:42.718832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:42.719043 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:42.719177 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:42.720515 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.721608 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.722875 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:42.723520 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:42.723618 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:42.727604 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:42.727616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:42.727634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:42.727674 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:42.731251 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.732076 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.732083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.732086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.732212 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:42.732631 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:42.732634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:42.732636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:42.732990 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:42.733218 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:42.733579 (+ 361us) server_negotiation.cc:299] Negotiation successful
0621 07:03:42.733854 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":344,"threads_started":1}
I20260621 07:03:42.735697 2284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:42.818500 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2161
I20260621 07:03:42.834110 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 1799
I20260621 07:03:42.835320 1861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:42.939240 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 1799
2026-06-21T07:03:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4177 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:03:43 dist-test-slave-hlq1 krb5kdc[2310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:03:43 dist-test-slave-hlq1 krb5kdc[2310](info): set up 2 sockets
Jun 21 07:03:43 dist-test-slave-hlq1 krb5kdc[2310](info): commencing operation
krb5kdc: starting...
W20260621 07:03:44.031652 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:03:44 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:03:44.210582 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39959
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:39959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:44.537279 2323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:44.537616 2323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:44.537781 2323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:44.544988 2323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:03:44.545143 2323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:44.545243 2323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:44.545354 2323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:03:44.545445 2323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:03:44.555152 2323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:39959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:44.556541 2323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:44.557828 2323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:44.569679 2328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:44.569772 2329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:44.569742 2331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:44.570904 2323 server_base.cc:1034] running on GCE node
I20260621 07:03:44.571728 2323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:44.573067 2323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:44.574306 2323 hybrid_clock.cc:648] HybridClock initialized: now 1782025424574266 us; error 36 us; skew 500 ppm
I20260621 07:03:44.574651 2323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:44 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025424, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:44.577805 2323 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:03:44.579355 2323 webserver.cc:460] Webserver started at http://127.1.188.254:41823/ using document root <none> and password file <none>
I20260621 07:03:44.580054 2323 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:44.580150 2323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:44.580412 2323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:44.583204 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55"
format_stamp: "Formatted at 2026-06-21 07:03:44 on dist-test-slave-hlq1"
server_key: "80029ef4b8991cac087563fc59c014db"
server_key_iv: "e62e6106cdc78d7598bc5bff1a0f3049"
server_key_version: "encryptionkey@0"
I20260621 07:03:44.583977 2323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55"
format_stamp: "Formatted at 2026-06-21 07:03:44 on dist-test-slave-hlq1"
server_key: "80029ef4b8991cac087563fc59c014db"
server_key_iv: "e62e6106cdc78d7598bc5bff1a0f3049"
server_key_version: "encryptionkey@0"
I20260621 07:03:44.589085 2323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:03:44.592406 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:44.593751 2323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:03:44.593947 2323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55"
format_stamp: "Formatted at 2026-06-21 07:03:44 on dist-test-slave-hlq1"
server_key: "80029ef4b8991cac087563fc59c014db"
server_key_iv: "e62e6106cdc78d7598bc5bff1a0f3049"
server_key_version: "encryptionkey@0"
I20260621 07:03:44.594152 2323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:44.609370 2323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:44.613710 2323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:44.614001 2323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:44.632241 2323 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:39959
I20260621 07:03:44.632290 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:39959 every 8 connection(s)
I20260621 07:03:44.633662 2323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:03:44.637456 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2323
I20260621 07:03:44.637697 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:03:44.637740 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:44.638348 1779 external_mini_cluster.cc:1428] Setting key aa28b4de92b33686225f49d673ea3ef1
I20260621 07:03:44.650480 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55: Bootstrap starting.
Jun 21 07:03:44 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:44.654023 2390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:44.655190 2390 log.cc:824] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:44.658465 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55: No bootstrap required, opened a new log
I20260621 07:03:44.660964 2393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:44.640289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.1.188.254:39959)
0621 07:03:44.641118 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:44.641130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:44.641183 (+ 53us) server_negotiation.cc:407] Connection header received
0621 07:03:44.642809 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:44.642834 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:44.643227 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:44.643709 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:44.645105 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:44.646351 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:44.647627 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:44.648335 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:44.648544 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:44.652887 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:44.652896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:44.652911 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:44.652952 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:44.656663 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:44.657427 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:44.657434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:44.657441 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:44.657610 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:44.658000 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:44.658004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:44.658006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:44.658549 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:44.658880 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:44.659382 (+ 502us) server_negotiation.cc:299] Negotiation successful
0621 07:03:44.659853 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":386,"threads_started":1}
I20260621 07:03:44.667022 2390 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } }
I20260621 07:03:44.667510 2390 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:44.667637 2390 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e67afcc01d4bbcb9eeea5cf9cb7a55, State: Initialized, Role: FOLLOWER
I20260621 07:03:44.668251 2390 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } }
I20260621 07:03:44.668496 2390 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:03:44.668633 2390 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:03:44.668820 2390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:44.671016 2390 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } }
I20260621 07:03:44.671648 2390 leader_election.cc:302] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7e67afcc01d4bbcb9eeea5cf9cb7a55; no voters:
I20260621 07:03:44.672305 2390 leader_election.cc:288] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:03:44.672461 2395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:44.673549 2395 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [term 1 LEADER]: Becoming Leader. State: Replica: d7e67afcc01d4bbcb9eeea5cf9cb7a55, State: Running, Role: LEADER
I20260621 07:03:44.674052 2395 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } }
I20260621 07:03:44.674515 2390 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:03:44.677030 2397 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7e67afcc01d4bbcb9eeea5cf9cb7a55. Latest consensus state: current_term: 1 leader_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } } }
I20260621 07:03:44.677490 2397 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:44.677284 2396 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7e67afcc01d4bbcb9eeea5cf9cb7a55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39959 } } }
I20260621 07:03:44.677681 2396 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55 [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:44.680092 2403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:03:44.686648 2403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:03:44.695263 2403 catalog_manager.cc:1347] Generated new cluster ID: 33ea3d2a184c4ad69263b39d724beda3
I20260621 07:03:44.695420 2403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:03:44.716161 2403 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:03:44.717572 2403 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:03:44.731328 2403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7e67afcc01d4bbcb9eeea5cf9cb7a55: Generated new TSK 0
I20260621 07:03:44.732451 2403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:03:44.775226 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:45.099906 2416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:45.100255 2416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:45.100431 2416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:45.107788 2416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:45.107954 2416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:45.108151 2416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:03:45.118312 2416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:45.119771 2416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:45.121182 2416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:45.134714 2422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:45.134631 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:45.134737 2424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:45.135680 2416 server_base.cc:1034] running on GCE node
I20260621 07:03:45.136351 2416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:45.137439 2416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:45.138679 2416 hybrid_clock.cc:648] HybridClock initialized: now 1782025425138627 us; error 76 us; skew 500 ppm
I20260621 07:03:45.139021 2416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:45 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025425, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:45.142351 2416 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:03:45.143999 2416 webserver.cc:460] Webserver started at http://127.1.188.193:40385/ using document root <none> and password file <none>
I20260621 07:03:45.144712 2416 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:45.144868 2416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:45.145185 2416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:45.148227 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "f755f0cc169349f9915b97e739b8598d"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "f6523dd79227ecd70631236ffd8cb69a"
server_key_iv: "04ef40a3e5442227a1683d4a30636002"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.149073 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "f755f0cc169349f9915b97e739b8598d"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "f6523dd79227ecd70631236ffd8cb69a"
server_key_iv: "04ef40a3e5442227a1683d4a30636002"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.154469 2416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:03:45.157994 2431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.159348 2416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:03:45.159569 2416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "f755f0cc169349f9915b97e739b8598d"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "f6523dd79227ecd70631236ffd8cb69a"
server_key_iv: "04ef40a3e5442227a1683d4a30636002"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.159783 2416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:45.182821 2416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:45.187026 2416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:45.187369 2416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:45.188444 2416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:45.189733 2416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:45.189821 2416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.189904 2416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:45.189947 2416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.223618 2416 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33043
I20260621 07:03:45.223742 2543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33043 every 8 connection(s)
I20260621 07:03:45.225009 2416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:03:45.234225 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2416
I20260621 07:03:45.234462 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:03:45.235239 1779 external_mini_cluster.cc:1428] Setting key dc7817fdb80dc6fd2c1b0945d7a69cb0
Jun 21 07:03:45 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025425, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:45.247056 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:45.227578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:43573 (local address 127.1.188.254:39959)
0621 07:03:45.228009 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:45.228013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:45.229007 (+ 994us) server_negotiation.cc:407] Connection header received
0621 07:03:45.230944 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:45.230955 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:45.231045 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:45.231178 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:45.232741 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.233480 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.235947 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.236473 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.236575 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:45.240223 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:45.240232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:45.240241 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:45.240290 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:45.243740 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:45.244732 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:45.244736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:45.244739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:45.244868 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:45.245352 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:45.245355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:45.245357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:45.245609 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:45.245714 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:45.246619 (+ 905us) server_negotiation.cc:299] Negotiation successful
0621 07:03:45.246798 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260621 07:03:45.248653 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:45.228156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39959 (local address 127.1.188.193:43573)
0621 07:03:45.228847 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:45.228893 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:45.230643 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:45.231467 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:45.231476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:45.231988 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:45.232562 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:45.232579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.233644 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.233648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:45.235772 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:45.235787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.236686 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.236693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:45.236973 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:45.237714 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:45.237737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:45.239974 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:45.243924 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:45.243936 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:45.243958 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:45.244607 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:45.244971 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:45.244979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:45.244986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:45.245234 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:45.245737 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:45.245748 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:45.246215 (+ 467us) client_negotiation.cc:769] Sending connection context
0621 07:03:45.246516 (+ 301us) client_negotiation.cc:240] Negotiation successful
0621 07:03:45.247106 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":260,"threads_started":1}
I20260621 07:03:45.250383 2544 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39959
I20260621 07:03:45.250759 2544 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:45.251729 2544 heartbeater.cc:511] Master 127.1.188.254:39959 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:03:45.254199 2355 ts_manager.cc:194] Registered new tserver with Master: f755f0cc169349f9915b97e739b8598d (127.1.188.193:33043)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
I20260621 07:03:45.256071 2355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:43573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:03:45.276535 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:45.596915 2550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:45.597239 2550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:45.597339 2550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:45.605147 2550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:45.605289 2550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:45.605406 2550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:03:45.615475 2550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:45.616852 2550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:45.618223 2550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:45.640396 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:45.640643 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:45.641176 2558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:45.642015 2550 server_base.cc:1034] running on GCE node
I20260621 07:03:45.642853 2550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:45.644095 2550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:45.645370 2550 hybrid_clock.cc:648] HybridClock initialized: now 1782025425645303 us; error 56 us; skew 500 ppm
I20260621 07:03:45.645743 2550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:45 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025425, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:45.649305 2550 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:03:45.651257 2550 webserver.cc:460] Webserver started at http://127.1.188.194:45419/ using document root <none> and password file <none>
I20260621 07:03:45.652240 2550 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:45.652395 2550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:45.652763 2550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:45.656108 2550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "3e41a3db38ed4a18a5b9e182420cf8e9"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "0654f6c6d0e9ec51eab364faa9e8be98"
server_key_iv: "197058ca2ae01cae905eea7174083395"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.656976 2550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "3e41a3db38ed4a18a5b9e182420cf8e9"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "0654f6c6d0e9ec51eab364faa9e8be98"
server_key_iv: "197058ca2ae01cae905eea7174083395"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.662696 2550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:03:45.666810 2565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.668628 2550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:03:45.668829 2550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "3e41a3db38ed4a18a5b9e182420cf8e9"
format_stamp: "Formatted at 2026-06-21 07:03:45 on dist-test-slave-hlq1"
server_key: "0654f6c6d0e9ec51eab364faa9e8be98"
server_key_iv: "197058ca2ae01cae905eea7174083395"
server_key_version: "encryptionkey@0"
I20260621 07:03:45.669063 2550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:45.706579 2550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:45.711457 2550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:45.711791 2550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:45.712873 2550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:45.714350 2550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:45.714464 2550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.714635 2550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:45.714723 2550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:45.751174 2550 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:33159
I20260621 07:03:45.751207 2677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:33159 every 8 connection(s)
I20260621 07:03:45.752620 2550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:03:45.759948 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2550
I20260621 07:03:45.760164 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:03:45.760739 1779 external_mini_cluster.cc:1428] Setting key 2c7edcecfac3c67bc0994ed083c294b2
Jun 21 07:03:45 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025425, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:45.777765 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:45.755461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:46229 (local address 127.1.188.254:39959)
0621 07:03:45.755957 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:45.755961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:45.756883 (+ 922us) server_negotiation.cc:407] Connection header received
0621 07:03:45.758759 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:45.758765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:45.758857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:45.758977 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:45.761210 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.762170 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.764929 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.765351 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.765464 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:45.770145 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:45.770156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:45.770164 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:45.770215 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:45.774013 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:45.774927 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:45.774936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:45.774942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:45.775125 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:45.775669 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:45.775677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:45.775682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:45.776083 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:45.776267 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:45.777011 (+ 744us) server_negotiation.cc:299] Negotiation successful
0621 07:03:45.777287 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":282,"threads_started":1}
I20260621 07:03:45.778524 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:45.756017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39959 (local address 127.1.188.194:46229)
0621 07:03:45.756716 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:45.756755 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:45.758472 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:45.759231 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:45.759245 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:45.760092 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:45.760943 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:45.760971 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.762402 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.762414 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:45.764737 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:45.764766 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:45.765614 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:45.765624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:45.766026 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:45.766847 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:45.766888 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:45.769851 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:45.774193 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:45.774201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:45.774215 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:45.774778 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:45.775265 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:45.775271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:45.775274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:45.775539 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:45.776242 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:45.776249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:45.776600 (+ 351us) client_negotiation.cc:769] Sending connection context
0621 07:03:45.776858 (+ 258us) client_negotiation.cc:240] Negotiation successful
0621 07:03:45.777309 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":286,"threads_started":1}
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:03:45.780157 2678 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39959
I20260621 07:03:45.780478 2678 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:45.781250 2678 heartbeater.cc:511] Master 127.1.188.254:39959 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:03:45.783109 2355 ts_manager.cc:194] Registered new tserver with Master: 3e41a3db38ed4a18a5b9e182420cf8e9 (127.1.188.194:33159)
I20260621 07:03:45.784453 2355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:46229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:03:45.806401 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:46.145815 2684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:46.146095 2684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:46.146263 2684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:46.153712 2684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:46.153824 2684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:46.154006 2684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:03:46.163858 2684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:46.165269 2684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:46.166625 2684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:46.180030 2692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:46.179955 2690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:46.180006 2689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:46.181012 2684 server_base.cc:1034] running on GCE node
I20260621 07:03:46.181691 2684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:46.182669 2684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:46.183866 2684 hybrid_clock.cc:648] HybridClock initialized: now 1782025426183817 us; error 44 us; skew 500 ppm
I20260621 07:03:46.184195 2684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:46.187279 2684 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:03:46.188853 2684 webserver.cc:460] Webserver started at http://127.1.188.195:40779/ using document root <none> and password file <none>
I20260621 07:03:46.189484 2684 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:46.189577 2684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:46.189814 2684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:46.192622 2684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "7eeee594fded4257b267a25dbbfd6991"
format_stamp: "Formatted at 2026-06-21 07:03:46 on dist-test-slave-hlq1"
server_key: "91aae71aa5fd0f70cee54ea261f30a76"
server_key_iv: "6d5030bf53b7e0e0f836dd3725f0b539"
server_key_version: "encryptionkey@0"
I20260621 07:03:46.193322 2684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "7eeee594fded4257b267a25dbbfd6991"
format_stamp: "Formatted at 2026-06-21 07:03:46 on dist-test-slave-hlq1"
server_key: "91aae71aa5fd0f70cee54ea261f30a76"
server_key_iv: "6d5030bf53b7e0e0f836dd3725f0b539"
server_key_version: "encryptionkey@0"
I20260621 07:03:46.198489 2684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 07:03:46.201891 2699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:46.203519 2684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:03:46.203712 2684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "7eeee594fded4257b267a25dbbfd6991"
format_stamp: "Formatted at 2026-06-21 07:03:46 on dist-test-slave-hlq1"
server_key: "91aae71aa5fd0f70cee54ea261f30a76"
server_key_iv: "6d5030bf53b7e0e0f836dd3725f0b539"
server_key_version: "encryptionkey@0"
I20260621 07:03:46.203903 2684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:46.227031 2684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:46.231377 2684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:46.231724 2684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:46.232659 2684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:03:46.233929 2684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:46.234031 2684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:46.234163 2684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:46.234228 2684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:46.259184 2544 heartbeater.cc:503] Master 127.1.188.254:39959 was elected leader, sending a full tablet report...
I20260621 07:03:46.267712 2684 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:44489
I20260621 07:03:46.267817 2811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:44489 every 8 connection(s)
I20260621 07:03:46.269068 2684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:03:46.275122 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2684
I20260621 07:03:46.275460 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:03:46.275938 1779 external_mini_cluster.cc:1428] Setting key bb80cd308fd7255ae4cf64884bd9205c
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:46.294811 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.271872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:38975 (local address 127.1.188.254:39959)
0621 07:03:46.272083 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.272087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.273275 (+ 1188us) server_negotiation.cc:407] Connection header received
0621 07:03:46.275454 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.275462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.275584 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.275701 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.277501 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.278509 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.280551 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.280964 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.281195 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.288342 (+ 7147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.288348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.288351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.288388 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.291590 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.292323 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.292330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.292334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.292459 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.292897 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.292903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.292906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.293208 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.293378 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.294234 (+ 856us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.294462 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:03:46.295905 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.272360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39959 (local address 127.1.188.195:38975)
0621 07:03:46.273095 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.273136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.275103 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.275946 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.275955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.276530 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:46.277347 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.277368 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.278704 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.278712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.280406 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.280418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.281256 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.281260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.281470 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.285815 (+ 4345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:46.285842 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:46.288112 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:46.291733 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:46.291742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:46.291760 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:46.292193 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:46.292572 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:46.292577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:46.292581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:46.292781 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:46.293344 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:46.293352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:46.293720 (+ 368us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.294120 (+ 400us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.294674 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":296,"threads_started":1}
I20260621 07:03:46.297360 2812 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39959
I20260621 07:03:46.297694 2812 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:46.298450 2812 heartbeater.cc:511] Master 127.1.188.254:39959 requested a full tablet report, sending...
I20260621 07:03:46.299962 2355 ts_manager.cc:194] Registered new tserver with Master: 7eeee594fded4257b267a25dbbfd6991 (127.1.188.195:44489)
I20260621 07:03:46.300750 2355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:38975
I20260621 07:03:46.304167 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:03:46.322351 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.310460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.1.188.254:39959)
0621 07:03:46.310621 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.310627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.311200 (+ 573us) server_negotiation.cc:407] Connection header received
0621 07:03:46.311321 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.311325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.311480 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.311570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.312973 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.313720 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.314886 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.315323 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.315506 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.317675 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.317681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.317684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.317715 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.320162 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.320729 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.320733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.320735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.320866 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.321175 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.321178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.321180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.321483 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.321664 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.321942 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.322089 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:03:46.328532 2355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:46.335306 2355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:46.359033 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.348188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.1.188.195:44489)
0621 07:03:46.349190 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.349203 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.350493 (+ 1290us) server_negotiation.cc:407] Connection header received
0621 07:03:46.350778 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.350785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.350958 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.351132 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:46.352574 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.354130 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.356441 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.357749 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.357882 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.358102 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.358468 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.358650 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":569,"threads_started":1}
I20260621 07:03:46.359875 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.347221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33043 (local address 127.0.0.1:59120)
0621 07:03:46.349722 (+ 2501us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.349766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.349931 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.351210 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.351218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.351253 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:46.351743 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.351755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.354062 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.354068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.356726 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.356739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.358500 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.358506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.358680 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.358720 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.358785 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.358969 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"mutex_wait_us":102,"thread_start_us":2912,"threads_started":1}
I20260621 07:03:46.360108 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.350793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.1.188.194:33159)
0621 07:03:46.352196 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.352206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.352349 (+ 143us) server_negotiation.cc:407] Connection header received
0621 07:03:46.352570 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.352577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.352732 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.352912 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:46.353401 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.354595 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.357289 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.358495 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.358630 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.358799 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.359634 (+ 835us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.359795 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":1256,"threads_started":1}
I20260621 07:03:46.360316 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.347435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.1.188.193:33043)
0621 07:03:46.350572 (+ 3137us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.350581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.350603 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:03:46.350704 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.350716 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.350953 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.351614 (+ 661us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:46.352285 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.353926 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.356896 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.358256 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.359372 (+ 1116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.359615 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.359710 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.359880 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2954,"thread_start_us":2814,"threads_started":1}
I20260621 07:03:46.360684 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.347599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44489 (local address 127.0.0.1:44092)
0621 07:03:46.350388 (+ 2789us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.350411 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.350595 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.351498 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.351502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.351521 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:46.352378 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.352392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.354309 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.354314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.356298 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.356309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.357933 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.357940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.358192 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.358250 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.359258 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.359423 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2672,"mutex_wait_us":1,"thread_start_us":1771,"threads_started":1}
I20260621 07:03:46.360765 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.351501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:33159 (local address 127.0.0.1:56718)
0621 07:03:46.352223 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.352247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.352400 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.352897 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.352901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.352924 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:46.353264 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.353272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.354765 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.354771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.357118 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.357130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.358948 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.358955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.359148 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.359261 (+ 113us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.359345 (+ 84us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.359467 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":39,"thread_start_us":392,"threads_started":1}
I20260621 07:03:46.364009 2479 tablet_service.cc:1467] Processing CreateTablet for tablet 339c9608586b41449d76058a9a19284d (DEFAULT_TABLE table=test-table [id=766800dcb9d0493d82dd20d049fd18c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:46.364009 2747 tablet_service.cc:1467] Processing CreateTablet for tablet 339c9608586b41449d76058a9a19284d (DEFAULT_TABLE table=test-table [id=766800dcb9d0493d82dd20d049fd18c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:46.363992 2613 tablet_service.cc:1467] Processing CreateTablet for tablet 339c9608586b41449d76058a9a19284d (DEFAULT_TABLE table=test-table [id=766800dcb9d0493d82dd20d049fd18c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:46.365260 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c9608586b41449d76058a9a19284d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:46.365391 2747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c9608586b41449d76058a9a19284d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:46.365397 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c9608586b41449d76058a9a19284d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:46.377636 2830 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Bootstrap starting.
I20260621 07:03:46.380494 2832 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Bootstrap starting.
I20260621 07:03:46.380977 2830 tablet_bootstrap.cc:654] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:46.382376 2830 log.cc:824] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:46.383777 2831 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Bootstrap starting.
I20260621 07:03:46.383985 2832 tablet_bootstrap.cc:654] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:46.385361 2830 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: No bootstrap required, opened a new log
I20260621 07:03:46.385427 2832 log.cc:824] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:46.385664 2830 ts_tablet_manager.cc:1397] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 07:03:46.388325 2831 tablet_bootstrap.cc:654] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:46.388417 2832 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: No bootstrap required, opened a new log
I20260621 07:03:46.388689 2832 ts_tablet_manager.cc:1397] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260621 07:03:46.389508 2831 log.cc:824] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:46.392443 2831 tablet_bootstrap.cc:492] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: No bootstrap required, opened a new log
I20260621 07:03:46.392705 2831 ts_tablet_manager.cc:1397] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 07:03:46.393726 2830 raft_consensus.cc:348] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.394074 2830 raft_consensus.cc:374] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:46.394174 2830 raft_consensus.cc:729] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e41a3db38ed4a18a5b9e182420cf8e9, State: Initialized, Role: FOLLOWER
I20260621 07:03:46.394742 2830 consensus_queue.cc:260] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.395893 2678 heartbeater.cc:503] Master 127.1.188.254:39959 was elected leader, sending a full tablet report...
I20260621 07:03:46.396342 2830 ts_tablet_manager.cc:1428] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260621 07:03:46.399609 2832 raft_consensus.cc:348] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.400032 2832 raft_consensus.cc:374] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:46.400130 2832 raft_consensus.cc:729] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f755f0cc169349f9915b97e739b8598d, State: Initialized, Role: FOLLOWER
I20260621 07:03:46.400709 2832 consensus_queue.cc:260] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.402385 2832 ts_tablet_manager.cc:1428] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260621 07:03:46.403875 2831 raft_consensus.cc:348] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.404197 2831 raft_consensus.cc:374] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:46.404284 2831 raft_consensus.cc:729] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeee594fded4257b267a25dbbfd6991, State: Initialized, Role: FOLLOWER
I20260621 07:03:46.404778 2831 consensus_queue.cc:260] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.405709 2812 heartbeater.cc:503] Master 127.1.188.254:39959 was elected leader, sending a full tablet report...
I20260621 07:03:46.406384 2831 ts_tablet_manager.cc:1428] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260621 07:03:46.477450 2545 tablet.cc:2369] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:03:46.504933 2679 tablet.cc:2369] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:03:46.520762 2813 tablet.cc:2369] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:03:46.533330 2836 raft_consensus.cc:482] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:46.533596 2836 raft_consensus.cc:504] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.535173 2836 leader_election.cc:288] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f755f0cc169349f9915b97e739b8598d (127.1.188.193:33043), 7eeee594fded4257b267a25dbbfd6991 (127.1.188.195:44489)
I20260621 07:03:46.544059 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.535683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:49895 (local address 127.1.188.193:33043)
0621 07:03:46.535885 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.535894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.537178 (+ 1284us) server_negotiation.cc:407] Connection header received
0621 07:03:46.537439 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.537446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.537516 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.537586 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:46.538206 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.539175 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.541661 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.542885 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.543044 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.543116 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.543673 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.543801 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:03:46.544056 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.535916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:58087 (local address 127.1.188.195:44489)
0621 07:03:46.536152 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.536162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.536835 (+ 673us) server_negotiation.cc:407] Connection header received
0621 07:03:46.537116 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.537121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.537202 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.537302 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:46.537870 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.538913 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.541677 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.542885 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.543044 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.543116 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.543681 (+ 565us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.543801 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:03:46.544353 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.535747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44489 (local address 127.1.188.194:58087)
0621 07:03:46.536721 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.536745 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.536955 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.537331 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.537338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.537365 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:46.537721 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.537733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.539050 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.539067 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.541512 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.541519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.543192 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.543200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.543477 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.543547 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.543642 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.543851 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":174,"thread_start_us":583,"threads_started":1}
I20260621 07:03:46.544378 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.535749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33043 (local address 127.1.188.194:49895)
0621 07:03:46.537054 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:46.537075 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:46.537222 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:46.537627 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:46.537633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:46.537663 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:46.538034 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.538041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.539297 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.539302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.541504 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:46.541518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.543194 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.543200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:46.543477 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.543537 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 07:03:46.543637 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:03:46.543851 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":310,"thread_start_us":782,"threads_started":1}
I20260621 07:03:46.545351 2767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c9608586b41449d76058a9a19284d" candidate_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eeee594fded4257b267a25dbbfd6991" is_pre_election: true
I20260621 07:03:46.545372 2499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c9608586b41449d76058a9a19284d" candidate_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f755f0cc169349f9915b97e739b8598d" is_pre_election: true
I20260621 07:03:46.545853 2767 raft_consensus.cc:2454] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e41a3db38ed4a18a5b9e182420cf8e9 in term 0.
I20260621 07:03:46.545877 2499 raft_consensus.cc:2454] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e41a3db38ed4a18a5b9e182420cf8e9 in term 0.
I20260621 07:03:46.546465 2569 leader_election.cc:302] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e41a3db38ed4a18a5b9e182420cf8e9, f755f0cc169349f9915b97e739b8598d; no voters:
I20260621 07:03:46.546876 2836 raft_consensus.cc:2781] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:46.546990 2836 raft_consensus.cc:482] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:46.547101 2836 raft_consensus.cc:3037] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:46.549607 2836 raft_consensus.cc:504] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.550294 2836 leader_election.cc:288] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [CANDIDATE]: Term 1 election: Requested vote from peers f755f0cc169349f9915b97e739b8598d (127.1.188.193:33043), 7eeee594fded4257b267a25dbbfd6991 (127.1.188.195:44489)
I20260621 07:03:46.550686 2499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c9608586b41449d76058a9a19284d" candidate_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f755f0cc169349f9915b97e739b8598d"
I20260621 07:03:46.550774 2767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c9608586b41449d76058a9a19284d" candidate_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eeee594fded4257b267a25dbbfd6991"
I20260621 07:03:46.550882 2499 raft_consensus.cc:3037] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:46.550998 2767 raft_consensus.cc:3037] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:46.553180 2499 raft_consensus.cc:2454] T 339c9608586b41449d76058a9a19284d P f755f0cc169349f9915b97e739b8598d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e41a3db38ed4a18a5b9e182420cf8e9 in term 1.
I20260621 07:03:46.553185 2767 raft_consensus.cc:2454] T 339c9608586b41449d76058a9a19284d P 7eeee594fded4257b267a25dbbfd6991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e41a3db38ed4a18a5b9e182420cf8e9 in term 1.
I20260621 07:03:46.553579 2569 leader_election.cc:302] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e41a3db38ed4a18a5b9e182420cf8e9, f755f0cc169349f9915b97e739b8598d; no voters:
I20260621 07:03:46.553859 2836 raft_consensus.cc:2781] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:46.554353 2836 raft_consensus.cc:686] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3e41a3db38ed4a18a5b9e182420cf8e9, State: Running, Role: LEADER
I20260621 07:03:46.554826 2836 consensus_queue.cc:237] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } }
I20260621 07:03:46.559572 2355 catalog_manager.cc:5495] T 339c9608586b41449d76058a9a19284d P 3e41a3db38ed4a18a5b9e182420cf8e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e41a3db38ed4a18a5b9e182420cf8e9 (127.1.188.194). New cstate: current_term: 1 leader_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f755f0cc169349f9915b97e739b8598d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e41a3db38ed4a18a5b9e182420cf8e9" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eeee594fded4257b267a25dbbfd6991" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44489 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:46.599884 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.585222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.1.188.193:33043)
0621 07:03:46.585441 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.585448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.585824 (+ 376us) server_negotiation.cc:407] Connection header received
0621 07:03:46.586036 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.586041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.586105 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.586217 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.587723 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.588520 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.589730 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.590143 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.590286 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.594199 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.594215 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.594232 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.594271 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.597148 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.597830 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.597838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.597840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.597988 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.598345 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.598350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.598354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.598709 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.599003 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.599300 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.599543 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:46.645586 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.630989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.1.188.193:33043)
0621 07:03:46.631265 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.631274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.631671 (+ 397us) server_negotiation.cc:407] Connection header received
0621 07:03:46.631881 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.631889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.631971 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.632076 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.633317 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.634398 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.635719 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.636366 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.636494 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.640466 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.640471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.640474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.640507 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.643085 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.643826 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.643829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.643832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.643957 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.644369 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.644373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.644375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.644623 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.644818 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.645091 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.645257 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:46.689729 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.675824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.1.188.193:33043)
0621 07:03:46.676074 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.676082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.676383 (+ 301us) server_negotiation.cc:407] Connection header received
0621 07:03:46.676612 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.676617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.676682 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.676784 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.678119 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.678897 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.680307 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.680728 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.680861 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.684660 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.684666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.684671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.684710 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.687207 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.687861 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.687868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.687872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.687991 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.688344 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.688349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.688352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.688638 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.688904 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.689178 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.689407 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:03:46.691560 2459 tablet_service.cc:2908] Rejecting scan request for tablet 339c9608586b41449d76058a9a19284d: Uninitialized: clean time has not yet been initialized
I20260621 07:03:46.694531 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2416
I20260621 07:03:46.707495 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.695533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.1.188.193:33043)
0621 07:03:46.695684 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.695687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.696051 (+ 364us) server_negotiation.cc:407] Connection header received
0621 07:03:46.696198 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.696201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.696266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.696344 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.697504 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.698638 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.699923 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.700316 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.700421 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.702591 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.702596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.702599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.702625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.705123 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.705792 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.705795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.705798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.705905 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.706218 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.706220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.706222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.706471 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.706678 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.707011 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.707200 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 07:03:46.708173 2539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:46.798795 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2416
I20260621 07:03:46.815371 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2550
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:03:46.830369 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.816184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.1.188.194:33159)
0621 07:03:46.816385 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.816393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.816415 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:03:46.816625 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.816630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.816715 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.816822 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.818025 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.818763 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.820224 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.820654 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.820791 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.824716 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.824736 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.824753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.824789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.827683 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.828332 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.828339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.828342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.828471 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.828852 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.828858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.828863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.829218 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.829531 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.829795 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.830028 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:03:46.831139 2673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:46.921468 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2550
I20260621 07:03:46.938601 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2684
Jun 21 07:03:46 dist-test-slave-hlq1 krb5kdc[2310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:03:46.952775 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:46.939452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.1.188.195:44489)
0621 07:03:46.939622 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:46.939627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:46.939646 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:03:46.939741 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:46.939745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:46.939817 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:46.939898 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:46.941073 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.941791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.943035 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:46.943482 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:46.943577 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:46.947570 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:46.947585 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:46.947602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:46.947634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:46.950275 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.950821 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.950828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.950830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.950949 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:46.951254 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:46.951257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:46.951259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:46.951716 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:46.951915 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:46.952247 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 07:03:46.952397 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:03:46.953405 2807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:47.043562 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2684
I20260621 07:03:47.061034 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2323
I20260621 07:03:47.062383 2385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:47.173698 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2323
2026-06-21T07:03:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4234 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:03:47 dist-test-slave-hlq1 krb5kdc[2860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:03:47 dist-test-slave-hlq1 krb5kdc[2860](info): set up 2 sockets
Jun 21 07:03:47 dist-test-slave-hlq1 krb5kdc[2860](info): commencing operation
krb5kdc: starting...
W20260621 07:03:48.260164 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:03:48 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:03:48.451299 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:36413
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:36413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:48.780337 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:48.780691 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:48.780857 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:48.788350 2873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:03:48.788496 2873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:03:48.788604 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:48.788733 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:48.788827 2873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:03:48.788920 2873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:03:48.799024 2873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:36413
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:48.800395 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:48.801949 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:48.813230 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:48.813284 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:48.813324 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:48.814451 2873 server_base.cc:1034] running on GCE node
I20260621 07:03:48.815183 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:48.816663 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:48.817878 2873 hybrid_clock.cc:648] HybridClock initialized: now 1782025428817820 us; error 31 us; skew 500 ppm
I20260621 07:03:48.818213 2873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:48 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025428, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:48.821405 2873 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:03:48.822940 2873 webserver.cc:460] Webserver started at http://127.1.188.254:37725/ using document root <none> and password file <none>
I20260621 07:03:48.823698 2873 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:48.823787 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:48.824054 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:48.826951 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "1bafb1152b45450f97a9cd186201c2eb"
format_stamp: "Formatted at 2026-06-21 07:03:48 on dist-test-slave-hlq1"
server_key: "ca62218b9f5d5d49dbac2e21bce9e1c9"
server_key_iv: "87d932c476a72cc6fc450c698dd1a2d2"
server_key_version: "encryptionkey@0"
I20260621 07:03:48.827795 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "1bafb1152b45450f97a9cd186201c2eb"
format_stamp: "Formatted at 2026-06-21 07:03:48 on dist-test-slave-hlq1"
server_key: "ca62218b9f5d5d49dbac2e21bce9e1c9"
server_key_iv: "87d932c476a72cc6fc450c698dd1a2d2"
server_key_version: "encryptionkey@0"
I20260621 07:03:48.832834 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:03:48.836316 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:48.837702 2873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:03:48.837883 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "1bafb1152b45450f97a9cd186201c2eb"
format_stamp: "Formatted at 2026-06-21 07:03:48 on dist-test-slave-hlq1"
server_key: "ca62218b9f5d5d49dbac2e21bce9e1c9"
server_key_iv: "87d932c476a72cc6fc450c698dd1a2d2"
server_key_version: "encryptionkey@0"
I20260621 07:03:48.838109 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:48.861683 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:48.865526 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:48.865800 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:48.888165 2873 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:36413
I20260621 07:03:48.888200 2949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:36413 every 8 connection(s)
I20260621 07:03:48.889583 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:03:48.893371 2950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:48.899314 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2873
I20260621 07:03:48.899559 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:03:48.900370 1779 external_mini_cluster.cc:1428] Setting key e0480ba1b5777763f186040b96c3cbe3
I20260621 07:03:48.905359 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Bootstrap starting.
I20260621 07:03:48.909039 2950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:48.910310 2950 log.cc:824] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:48.913868 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: No bootstrap required, opened a new log
Jun 21 07:03:48 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:48.922392 2950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } }
I20260621 07:03:48.922811 2950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:48.922958 2950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bafb1152b45450f97a9cd186201c2eb, State: Initialized, Role: FOLLOWER
I20260621 07:03:48.923710 2950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } }
I20260621 07:03:48.924037 2950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:03:48.924214 2950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:03:48.924397 2950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:48.924701 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:48.902628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.1.188.254:36413)
0621 07:03:48.903553 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:48.903566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:48.903610 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:03:48.905240 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:48.905263 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:48.905703 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:48.906033 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:48.907848 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:48.909164 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:48.910930 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:48.911962 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:48.912095 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:48.916986 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:48.916995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:48.917016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:48.917061 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:48.920308 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:48.921036 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:48.921044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:48.921050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:48.921228 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:48.921613 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:48.921618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:48.921621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:48.922176 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:48.922481 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:48.922961 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 07:03:48.923482 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":423,"threads_started":1}
I20260621 07:03:48.926827 2950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } }
I20260621 07:03:48.927505 2950 leader_election.cc:302] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bafb1152b45450f97a9cd186201c2eb; no voters:
I20260621 07:03:48.928171 2950 leader_election.cc:288] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:03:48.928341 2955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:48.929082 2955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [term 1 LEADER]: Becoming Leader. State: Replica: 1bafb1152b45450f97a9cd186201c2eb, State: Running, Role: LEADER
I20260621 07:03:48.929632 2955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } }
I20260621 07:03:48.930328 2950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:03:48.932324 2957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bafb1152b45450f97a9cd186201c2eb. Latest consensus state: current_term: 1 leader_uuid: "1bafb1152b45450f97a9cd186201c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } } }
I20260621 07:03:48.932516 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bafb1152b45450f97a9cd186201c2eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bafb1152b45450f97a9cd186201c2eb" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 36413 } } }
I20260621 07:03:48.932778 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:48.932778 2957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:48.933786 2962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:03:48.940193 2962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:03:48.951124 2962 catalog_manager.cc:1347] Generated new cluster ID: da62224ec7014b018ce449aca2ab836c
I20260621 07:03:48.951253 2962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:03:48.968416 2962 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:03:48.969944 2962 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:03:48.976024 2962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Generated new TSK 0
I20260621 07:03:48.977073 2962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:03:49.019825 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:36413
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:49.343494 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:49.343767 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:49.343950 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:49.351377 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:49.351532 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:49.351718 2976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:49.351854 2976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:03:49.361733 2976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:36413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:49.363016 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:49.364387 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:49.377499 2982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:49.377427 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:49.377986 2981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:49.378404 2976 server_base.cc:1034] running on GCE node
I20260621 07:03:49.379060 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:49.380198 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:49.381467 2976 hybrid_clock.cc:648] HybridClock initialized: now 1782025429381375 us; error 73 us; skew 500 ppm
I20260621 07:03:49.381817 2976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:49 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025429, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:49.385270 2976 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:03:49.386775 2976 webserver.cc:460] Webserver started at http://127.1.188.193:44813/ using document root <none> and password file <none>
I20260621 07:03:49.387516 2976 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:49.387609 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:49.387995 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:49.390884 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "4f8f8d0bea38465b8e35a63bc878f941"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "5f4df6e3c25d8a1ef3082a160a0e14b2"
server_key_iv: "148e98cd9477f43a890e42b95483bf37"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.391714 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "4f8f8d0bea38465b8e35a63bc878f941"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "5f4df6e3c25d8a1ef3082a160a0e14b2"
server_key_iv: "148e98cd9477f43a890e42b95483bf37"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.396936 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:03:49.400452 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:49.401835 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:03:49.402000 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "4f8f8d0bea38465b8e35a63bc878f941"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "5f4df6e3c25d8a1ef3082a160a0e14b2"
server_key_iv: "148e98cd9477f43a890e42b95483bf37"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.402223 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:49.420218 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:49.424595 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:49.424870 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:49.427666 2976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:49.427773 2976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:49.427906 2976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:49.427977 2976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:49 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025429, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:49.451545 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.428383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45979 (local address 127.1.188.254:36413)
0621 07:03:49.429035 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:49.429040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:49.430001 (+ 961us) server_negotiation.cc:407] Connection header received
0621 07:03:49.432562 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:49.432567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:49.432671 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:49.432789 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:49.434582 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.435563 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.438209 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.438800 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.438995 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.444559 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:49.444568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:49.444574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:49.444618 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:49.448101 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.449037 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.449045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.449051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.449200 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.449733 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.449739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.449743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.450027 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:49.450184 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:49.450893 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 07:03:49.451123 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":432,"threads_started":1}
I20260621 07:03:49.452781 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.429036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.193:45979)
0621 07:03:49.429833 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:49.429886 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:49.432266 (+ 2380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:49.433030 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:49.433038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:49.433670 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:49.434408 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.434439 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.435769 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.435779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.438037 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.438049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.439092 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.439101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.439507 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.440508 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:49.440545 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:49.444279 (+ 3734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:49.448252 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.448261 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.448277 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.448836 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.449324 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.449329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.449332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.449616 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.450172 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:49.450180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:49.450545 (+ 365us) client_negotiation.cc:769] Sending connection context
0621 07:03:49.450815 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 07:03:49.451386 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":342,"threads_started":1}
I20260621 07:03:49.479483 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.459997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.193:47101)
0621 07:03:49.463938 (+ 3941us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:49.463959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:49.464164 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:49.464604 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:49.464612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:49.465346 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:49.465837 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.465855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.466887 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.466895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.469089 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.469108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.470607 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.470616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.470869 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.471800 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:49.471844 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:49.472830 (+ 986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:49.476707 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.476713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.476717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.477147 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.477529 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.477534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.477538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.477678 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.478245 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:49.478249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:49.478459 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 07:03:49.478615 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 07:03:49.478938 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3771,"thread_start_us":3619,"threads_started":1}
I20260621 07:03:49.479664 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.460342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:47101 (local address 127.1.188.254:36413)
0621 07:03:49.460586 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:49.460601 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:49.464073 (+ 3472us) server_negotiation.cc:407] Connection header received
0621 07:03:49.464319 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:49.464324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:49.464419 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:49.464529 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:49.466038 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.466735 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.469277 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.470095 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.470213 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.473021 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:49.473030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:49.473037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:49.473081 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:49.476566 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.477276 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.477281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.477285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.477418 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.477833 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.477842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.477846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.478109 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:49.478270 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:49.478829 (+ 559us) server_negotiation.cc:299] Negotiation successful
0621 07:03:49.479155 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:03:49.486011 2976 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:42097
I20260621 07:03:49.486029 3111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:42097 every 8 connection(s)
I20260621 07:03:49.487272 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:03:49.489625 3112 heartbeater.cc:347] Connected to a master server at 127.1.188.254:36413
I20260621 07:03:49.489750 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 2976
I20260621 07:03:49.489864 3112 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:49.489992 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:03:49.490624 1779 external_mini_cluster.cc:1428] Setting key 7567dcc9e877a034d922003c20243e98
I20260621 07:03:49.490648 3112 heartbeater.cc:511] Master 127.1.188.254:36413 requested a full tablet report, sending...
I20260621 07:03:49.492827 2904 ts_manager.cc:194] Registered new tserver with Master: 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193:42097)
I20260621 07:03:49.494246 2904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:45979
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:03:49.530134 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:36413
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:49.848253 3116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:49.848577 3116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:49.848752 3116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:49.855964 3116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:49.856101 3116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:49.856277 3116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:49.856423 3116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:03:49.866276 3116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:36413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:49.867657 3116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:49.869022 3116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:49.882457 3121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:49.882424 3122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:49.882495 3124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:49.883327 3116 server_base.cc:1034] running on GCE node
I20260621 07:03:49.884016 3116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:49.885063 3116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:49.886399 3116 hybrid_clock.cc:648] HybridClock initialized: now 1782025429886303 us; error 48 us; skew 500 ppm
I20260621 07:03:49.886739 3116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:49 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025429, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:49.890295 3116 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:03:49.892053 3116 webserver.cc:460] Webserver started at http://127.1.188.194:39413/ using document root <none> and password file <none>
I20260621 07:03:49.892765 3116 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:49.892895 3116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:49.893220 3116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:49.896157 3116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "0daba67ffee44545b332ba05af07fbd5"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "619e54c99b2acb3a234658bea1e19a14"
server_key_iv: "1e1aebb103d881b5f76eb2d9e6fbc73a"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.897097 3116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "0daba67ffee44545b332ba05af07fbd5"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "619e54c99b2acb3a234658bea1e19a14"
server_key_iv: "1e1aebb103d881b5f76eb2d9e6fbc73a"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.902451 3116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:03:49.905902 3131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:49.907335 3116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:03:49.907546 3116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "0daba67ffee44545b332ba05af07fbd5"
format_stamp: "Formatted at 2026-06-21 07:03:49 on dist-test-slave-hlq1"
server_key: "619e54c99b2acb3a234658bea1e19a14"
server_key_iv: "1e1aebb103d881b5f76eb2d9e6fbc73a"
server_key_version: "encryptionkey@0"
I20260621 07:03:49.907789 3116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:49.929251 3116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:49.933861 3116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:49.934154 3116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:49.937152 3116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:49.937258 3116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:49.937446 3116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:49.937553 3116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:49 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025429, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:49.961696 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.937603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:42893 (local address 127.1.188.254:36413)
0621 07:03:49.937790 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:49.937815 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:49.939338 (+ 1523us) server_negotiation.cc:407] Connection header received
0621 07:03:49.941499 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:49.941505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:49.941630 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:49.941755 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:49.944408 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.945369 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.948848 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.949318 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.949460 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.954014 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:49.954019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:49.954022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:49.954057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:49.957871 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.958682 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.958691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.958698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.958902 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.959420 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.959427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.959433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.959873 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:49.960046 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:49.960869 (+ 823us) server_negotiation.cc:299] Negotiation successful
0621 07:03:49.961177 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:03:49.962301 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.938304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.194:42893)
0621 07:03:49.939153 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:49.939202 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:49.941201 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:49.942069 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:49.942086 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:49.943049 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:49.944184 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.944222 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.945590 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.945597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.948656 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.948677 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.949571 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.949579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.949819 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.950534 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:49.950555 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:49.953767 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:49.958071 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.958081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.958108 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.958525 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.959034 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.959038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.959040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.959275 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.960032 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:49.960043 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:49.960474 (+ 431us) client_negotiation.cc:769] Sending connection context
0621 07:03:49.960707 (+ 233us) client_negotiation.cc:240] Negotiation successful
0621 07:03:49.961106 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":363,"threads_started":1}
I20260621 07:03:49.983729 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.968736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:35163 (local address 127.1.188.254:36413)
0621 07:03:49.968970 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:49.968978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:49.969537 (+ 559us) server_negotiation.cc:407] Connection header received
0621 07:03:49.969687 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:49.969695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:49.969807 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:49.969937 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:49.970861 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.972399 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.973701 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.974444 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.974585 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.976900 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:49.976909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:49.976917 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:49.976966 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:49.980618 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.981421 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.981428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.981431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.981585 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:49.982073 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:49.982081 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:49.982084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:49.982562 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:49.982726 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:49.983139 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:03:49.983346 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:03:49.983819 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:49.968806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.194:35163)
0621 07:03:49.969437 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:49.969453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:49.969561 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:49.969966 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:49.969969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:49.970434 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:49.970721 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.970730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.972550 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.972557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.973551 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:49.973561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:49.974689 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:49.974698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:49.974959 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:49.975666 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:49.975696 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:49.976687 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:49.980751 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.980757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.980761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.981237 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.981758 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:49.981765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:49.981770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:49.981946 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:49.982690 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:49.982695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:49.982887 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 07:03:49.983030 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 07:03:49.983346 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":337,"threads_started":1}
I20260621 07:03:49.997888 3116 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:38581
I20260621 07:03:49.997906 3250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:38581 every 8 connection(s)
I20260621 07:03:49.999195 3116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:03:50.001487 3251 heartbeater.cc:347] Connected to a master server at 127.1.188.254:36413
I20260621 07:03:50.001721 3251 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:50.002354 3251 heartbeater.cc:511] Master 127.1.188.254:36413 requested a full tablet report, sending...
I20260621 07:03:50.003769 2904 ts_manager.cc:194] Registered new tserver with Master: 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581)
I20260621 07:03:50.004627 2904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:42893
I20260621 07:03:50.008759 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3116
I20260621 07:03:50.008916 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:03:50.009310 1779 external_mini_cluster.cc:1428] Setting key 4bb47ee3b100e110096c72948bcbb03e
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:03:50.048731 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:36413
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:50.370627 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:50.370918 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:50.371061 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:50.378368 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:50.378477 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:50.378634 3255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:50.378748 3255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:03:50.388641 3255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:36413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:50.390458 3255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:50.391949 3255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:50.405402 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:50.405347 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:50.405588 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:50.406479 3255 server_base.cc:1034] running on GCE node
I20260621 07:03:50.407099 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:50.408138 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:50.409354 3255 hybrid_clock.cc:648] HybridClock initialized: now 1782025430409280 us; error 51 us; skew 500 ppm
I20260621 07:03:50.409655 3255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:50 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:50.413015 3255 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:03:50.414408 3255 webserver.cc:460] Webserver started at http://127.1.188.195:46339/ using document root <none> and password file <none>
I20260621 07:03:50.415131 3255 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:50.415262 3255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:50.415640 3255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:50.418522 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "babf607b935d487e84012f218b6489bd"
format_stamp: "Formatted at 2026-06-21 07:03:50 on dist-test-slave-hlq1"
server_key: "52a257d02247fd9e0ed9c227efbc6b42"
server_key_iv: "2aa4f8c02699ed5a3da489f0b679f807"
server_key_version: "encryptionkey@0"
I20260621 07:03:50.419319 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "babf607b935d487e84012f218b6489bd"
format_stamp: "Formatted at 2026-06-21 07:03:50 on dist-test-slave-hlq1"
server_key: "52a257d02247fd9e0ed9c227efbc6b42"
server_key_iv: "2aa4f8c02699ed5a3da489f0b679f807"
server_key_version: "encryptionkey@0"
I20260621 07:03:50.424335 3255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:03:50.427809 3270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:50.429056 3255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:03:50.429270 3255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "babf607b935d487e84012f218b6489bd"
format_stamp: "Formatted at 2026-06-21 07:03:50 on dist-test-slave-hlq1"
server_key: "52a257d02247fd9e0ed9c227efbc6b42"
server_key_iv: "2aa4f8c02699ed5a3da489f0b679f807"
server_key_version: "encryptionkey@0"
I20260621 07:03:50.429471 3255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:50.441380 3255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:50.445453 3255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:50.445727 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:50.448474 3255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:50.448585 3255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:50.448768 3255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:50.448841 3255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:50 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:50.471938 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.448840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:45563 (local address 127.1.188.254:36413)
0621 07:03:50.449009 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.449013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.450394 (+ 1381us) server_negotiation.cc:407] Connection header received
0621 07:03:50.452521 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.452528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.452644 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.452770 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:50.454964 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.456101 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.458548 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.459150 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.459314 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.465050 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:50.465058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:50.465064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:50.465114 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:50.468168 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.469195 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.469202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.469208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.469359 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.469886 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.469893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.469898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.470178 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:50.470344 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.471304 (+ 960us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.471573 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:03:50.473536 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.449471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.195:45563)
0621 07:03:50.450224 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.450278 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.452160 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.453036 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.453050 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.453816 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:50.454784 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.454806 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.456317 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.456325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.458386 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.458398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.459447 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.459455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.459757 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.460719 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:50.460756 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:50.464719 (+ 3963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:50.468305 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.468318 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.468338 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.468987 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.469503 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.469510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.469519 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.469732 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.470315 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:50.470322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:50.470803 (+ 481us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.471204 (+ 401us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.471815 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":327,"threads_started":1}
I20260621 07:03:50.497852 3112 heartbeater.cc:503] Master 127.1.188.254:36413 was elected leader, sending a full tablet report...
I20260621 07:03:50.498937 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.481036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:58547 (local address 127.1.188.254:36413)
0621 07:03:50.481205 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.481209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.481770 (+ 561us) server_negotiation.cc:407] Connection header received
0621 07:03:50.482013 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.482018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.482109 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.482218 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:50.483724 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.484974 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.487808 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.488460 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.488626 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.491355 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:50.491365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:50.491372 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:50.491446 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:50.494769 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.495831 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.495839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.495844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.496048 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.496469 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.496474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.496477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.496756 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:50.496912 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.498343 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.498570 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:03:50.499315 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.481069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.1.188.195:58547)
0621 07:03:50.481662 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.481681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.481867 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.482259 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.482269 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.483004 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:50.483491 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.483508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.485146 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.485153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.487583 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.487615 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.488708 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.488718 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.489000 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.489735 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:50.489762 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:50.491033 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:50.494931 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.494940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.494947 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.495691 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.496191 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.496204 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.496206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.496330 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.496875 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:50.496880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:50.497116 (+ 236us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.497311 (+ 195us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.497698 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":345,"threads_started":1}
I20260621 07:03:50.506809 3255 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:41911
I20260621 07:03:50.506840 3389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:41911 every 8 connection(s)
I20260621 07:03:50.508040 3255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:03:50.510115 3390 heartbeater.cc:347] Connected to a master server at 127.1.188.254:36413
I20260621 07:03:50.510344 3390 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:50.510999 3390 heartbeater.cc:511] Master 127.1.188.254:36413 requested a full tablet report, sending...
I20260621 07:03:50.512441 2904 ts_manager.cc:194] Registered new tserver with Master: babf607b935d487e84012f218b6489bd (127.1.188.195:41911)
I20260621 07:03:50.513276 2904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:45563
I20260621 07:03:50.516549 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3255
I20260621 07:03:50.516742 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:03:50.517194 1779 external_mini_cluster.cc:1428] Setting key 78887dfa086dd7b424f3e80dc5964168
I20260621 07:03:50.520459 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:03:50 dist-test-slave-hlq1 krb5kdc[2860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:03:50 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:50.566289 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.552101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.1.188.254:36413)
0621 07:03:50.552349 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.552357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.552477 (+ 120us) server_negotiation.cc:407] Connection header received
0621 07:03:50.552629 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.552633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.552699 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.552788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:50.554195 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.554955 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.556402 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.556983 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.557141 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.561012 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:50.561017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:50.561020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:50.561054 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:50.563724 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.564380 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.564384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.564386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.564511 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.564945 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.564950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.564953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.565233 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:50.565492 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.565727 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.565880 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:03:50.570582 2904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:50.577720 2904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:50.598999 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.590213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.1.188.195:41911)
0621 07:03:50.592828 (+ 2615us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.592835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.592860 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:03:50.592932 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.592938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.593113 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.593328 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:50.593978 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.595018 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.596778 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.597706 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.597854 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.598087 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.598453 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.598608 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2475,"thread_start_us":2320,"threads_started":1}
I20260621 07:03:50.599686 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.589995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:41911 (local address 127.0.0.1:58934)
0621 07:03:50.591120 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.591161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.591312 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.593330 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.593337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.593366 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:50.593763 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.593779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.595146 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.595155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.596632 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.596643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.597911 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.597918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.598127 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.598175 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.598317 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.598501 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":734,"threads_started":1}
I20260621 07:03:50.602924 3320 tablet_service.cc:1467] Processing CreateTablet for tablet 5c810bc59dfb4428ad82db5435d805ec (DEFAULT_TABLE table=test-table [id=d029fc34630a449abffd6bb2b0df3db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.604266 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c810bc59dfb4428ad82db5435d805ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.611490 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.590203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42097 (local address 127.0.0.1:59560)
0621 07:03:50.592081 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.592103 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.592236 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.604294 (+ 12058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.604300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.604329 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:50.604655 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.604668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.606599 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.606605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.608729 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.608745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.610631 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.610637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.610804 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.610841 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.610912 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.611048 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1709,"thread_start_us":1552,"threads_started":1}
I20260621 07:03:50.612082 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.592089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:38581 (local address 127.0.0.1:48714)
0621 07:03:50.599938 (+ 7849us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.599954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.600047 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.606111 (+ 6064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.606116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.606141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:50.606372 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.606383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.608010 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.608016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.609555 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.609568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.611329 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.611335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.611553 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.611586 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.611675 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.611809 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7742,"thread_start_us":13549,"threads_started":1}
I20260621 07:03:50.612643 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.590782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.1.188.193:42097)
0621 07:03:50.603614 (+ 12832us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.603626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.603660 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:03:50.603767 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.603777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.604042 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.604277 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:50.604870 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.606452 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.608897 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.610399 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.611951 (+ 1552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.612190 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.612264 (+ 74us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.612409 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12623,"thread_start_us":14949,"threads_started":1}
I20260621 07:03:50.612941 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.591795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.1.188.194:38581)
0621 07:03:50.604878 (+ 13083us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.604889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.604924 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:03:50.605036 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.605046 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.605320 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.606542 (+ 1222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:50.606744 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.607869 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.609750 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.611119 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.611257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.611479 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.612472 (+ 993us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.612641 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12922,"thread_start_us":12818,"threads_started":1}
I20260621 07:03:50.616389 3042 tablet_service.cc:1467] Processing CreateTablet for tablet 5c810bc59dfb4428ad82db5435d805ec (DEFAULT_TABLE table=test-table [id=d029fc34630a449abffd6bb2b0df3db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.616389 3181 tablet_service.cc:1467] Processing CreateTablet for tablet 5c810bc59dfb4428ad82db5435d805ec (DEFAULT_TABLE table=test-table [id=d029fc34630a449abffd6bb2b0df3db2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.617810 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c810bc59dfb4428ad82db5435d805ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.617789 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c810bc59dfb4428ad82db5435d805ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.617990 3409 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Bootstrap starting.
I20260621 07:03:50.621654 3409 tablet_bootstrap.cc:654] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.622632 3409 log.cc:824] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:50.625574 3409 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: No bootstrap required, opened a new log
I20260621 07:03:50.625916 3409 ts_tablet_manager.cc:1397] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260621 07:03:50.633881 3412 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Bootstrap starting.
I20260621 07:03:50.633988 3411 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Bootstrap starting.
I20260621 07:03:50.633809 3409 raft_consensus.cc:348] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.634521 3409 raft_consensus.cc:374] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.634634 3409 raft_consensus.cc:729] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babf607b935d487e84012f218b6489bd, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.635237 3409 consensus_queue.cc:260] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.636413 3390 heartbeater.cc:503] Master 127.1.188.254:36413 was elected leader, sending a full tablet report...
I20260621 07:03:50.637099 3409 ts_tablet_manager.cc:1428] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260621 07:03:50.637395 3412 tablet_bootstrap.cc:654] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.637660 3411 tablet_bootstrap.cc:654] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.638648 3411 log.cc:824] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:50.638697 3412 log.cc:824] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:50.641458 3412 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: No bootstrap required, opened a new log
I20260621 07:03:50.641686 3411 tablet_bootstrap.cc:492] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: No bootstrap required, opened a new log
I20260621 07:03:50.641806 3412 ts_tablet_manager.cc:1397] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260621 07:03:50.641961 3411 ts_tablet_manager.cc:1397] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 07:03:50.649163 3412 raft_consensus.cc:348] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.649542 3412 raft_consensus.cc:374] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.649680 3412 raft_consensus.cc:729] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daba67ffee44545b332ba05af07fbd5, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.649449 3411 raft_consensus.cc:348] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.649941 3411 raft_consensus.cc:374] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.650049 3411 raft_consensus.cc:729] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f8d0bea38465b8e35a63bc878f941, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.650357 3412 consensus_queue.cc:260] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.650663 3411 consensus_queue.cc:260] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.651548 3251 heartbeater.cc:503] Master 127.1.188.254:36413 was elected leader, sending a full tablet report...
I20260621 07:03:50.652318 3412 ts_tablet_manager.cc:1428] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260621 07:03:50.652433 3411 ts_tablet_manager.cc:1428] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260621 07:03:50.667904 3417 raft_consensus.cc:482] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:50.668203 3417 raft_consensus.cc:504] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.669911 3417 leader_election.cc:288] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers babf607b935d487e84012f218b6489bd (127.1.188.195:41911), 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581)
I20260621 07:03:50.676843 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.670538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:40587 (local address 127.1.188.194:38581)
0621 07:03:50.670793 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.670802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.671507 (+ 705us) server_negotiation.cc:407] Connection header received
0621 07:03:50.671709 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.671712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.671771 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.671863 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:50.672489 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.673157 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.674925 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.675837 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.676040 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.676118 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.676456 (+ 338us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.676580 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:03:50.676860 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.670369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:35345 (local address 127.1.188.195:41911)
0621 07:03:50.670642 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.670652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.671298 (+ 646us) server_negotiation.cc:407] Connection header received
0621 07:03:50.671568 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.671571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.671634 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.671730 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:50.672316 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.673026 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.674957 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.675866 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.676067 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.676155 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.676458 (+ 303us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.676585 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:03:50.677032 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.670301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:41911 (local address 127.1.188.193:35345)
0621 07:03:50.671181 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.671200 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.671379 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.671746 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.671750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.671775 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:50.672166 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.672178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.673167 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.673175 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.674819 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.674830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.676115 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.676122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.676298 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.676326 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.676431 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.676621 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":77,"spinlock_wait_cycles":17792,"thread_start_us":711,"threads_started":1}
I20260621 07:03:50.677448 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.670370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:38581 (local address 127.1.188.193:40587)
0621 07:03:50.671365 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.671379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.671567 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.671891 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.671895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.671936 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:50.672344 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.672353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.673304 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.673309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.674779 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.674790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.676082 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.676092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.676294 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.676322 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.676428 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.676560 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":40,"thread_start_us":483,"threads_started":1}
I20260621 07:03:50.678197 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c810bc59dfb4428ad82db5435d805ec" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd" is_pre_election: true
I20260621 07:03:50.678531 3206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c810bc59dfb4428ad82db5435d805ec" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daba67ffee44545b332ba05af07fbd5" is_pre_election: true
I20260621 07:03:50.678716 3340 raft_consensus.cc:2454] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 0.
I20260621 07:03:50.679026 3206 raft_consensus.cc:2454] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 0.
I20260621 07:03:50.679459 2992 leader_election.cc:302] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f8d0bea38465b8e35a63bc878f941, babf607b935d487e84012f218b6489bd; no voters:
I20260621 07:03:50.679926 3417 raft_consensus.cc:2781] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:50.680217 3417 raft_consensus.cc:482] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:50.680347 3417 raft_consensus.cc:3037] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.682880 3417 raft_consensus.cc:504] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.683660 3417 leader_election.cc:288] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 election: Requested vote from peers babf607b935d487e84012f218b6489bd (127.1.188.195:41911), 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581)
I20260621 07:03:50.684027 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c810bc59dfb4428ad82db5435d805ec" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd"
I20260621 07:03:50.684130 3206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c810bc59dfb4428ad82db5435d805ec" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daba67ffee44545b332ba05af07fbd5"
I20260621 07:03:50.684253 3340 raft_consensus.cc:3037] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.684376 3206 raft_consensus.cc:3037] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.686550 3206 raft_consensus.cc:2454] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 1.
I20260621 07:03:50.686554 3340 raft_consensus.cc:2454] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 1.
I20260621 07:03:50.686990 2992 leader_election.cc:302] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f8d0bea38465b8e35a63bc878f941, babf607b935d487e84012f218b6489bd; no voters:
I20260621 07:03:50.687294 3417 raft_consensus.cc:2781] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:50.687777 3417 raft_consensus.cc:686] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8f8d0bea38465b8e35a63bc878f941, State: Running, Role: LEADER
I20260621 07:03:50.688265 3417 consensus_queue.cc:237] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } }
I20260621 07:03:50.693049 2904 catalog_manager.cc:5495] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193). New cstate: current_term: 1 leader_uuid: "4f8f8d0bea38465b8e35a63bc878f941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } health_report { overall_health: UNKNOWN } } }
I20260621 07:03:50.721378 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.715742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.1.188.193:42097)
0621 07:03:50.715938 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.715945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.716076 (+ 131us) server_negotiation.cc:407] Connection header received
0621 07:03:50.716425 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.716429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.716526 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.716675 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:50.717058 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.717762 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.719114 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.719581 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.719711 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.720232 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:50.720757 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:50.720915 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.721014 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.721107 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 07:03:50.729871 3340 raft_consensus.cc:1261] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Refusing update from remote peer 4f8f8d0bea38465b8e35a63bc878f941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:50.729868 3206 raft_consensus.cc:1261] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Refusing update from remote peer 4f8f8d0bea38465b8e35a63bc878f941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:50.730928 3420 consensus_queue.cc:1035] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:50.731317 3417 consensus_queue.cc:1035] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:03:50.739696 3113 tablet.cc:2369] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:03:50.742702 3425 mvcc.cc:205] Tried to move back new op lower bound from 7299176164259262464 to 7299176164102660096. Current Snapshot: MvccSnapshot[applied={T|T < 7299176164259262464}]
I20260621 07:03:50.762879 2904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:50.763151 2904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:50.766906 2904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Sending DeleteTablet for 3 replicas of tablet 5c810bc59dfb4428ad82db5435d805ec
I20260621 07:03:50.768186 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c810bc59dfb4428ad82db5435d805ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:50 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58934
I20260621 07:03:50.768622 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c810bc59dfb4428ad82db5435d805ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:50 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59560
I20260621 07:03:50.768945 3181 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c810bc59dfb4428ad82db5435d805ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:50 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:48714
I20260621 07:03:50.770107 2904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:50.771003 2904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:50.772914 3435 tablet_replica.cc:330] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: stopping tablet replica
I20260621 07:03:50.773389 3435 raft_consensus.cc:2229] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:50.773924 3435 raft_consensus.cc:2258] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:50.775002 3434 tablet_replica.cc:330] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: stopping tablet replica
I20260621 07:03:50.777595 3435 ts_tablet_manager.cc:1905] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:50.780151 3181 tablet_service.cc:1467] Processing CreateTablet for tablet 86b14fc0048141e280970f10a194f0ed (DEFAULT_TABLE table=test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.780179 3319 tablet_service.cc:1467] Processing CreateTablet for tablet 86b14fc0048141e280970f10a194f0ed (DEFAULT_TABLE table=test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.780599 3319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b14fc0048141e280970f10a194f0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.780611 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b14fc0048141e280970f10a194f0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.784011 3042 tablet_service.cc:1467] Processing CreateTablet for tablet 86b14fc0048141e280970f10a194f0ed (DEFAULT_TABLE table=test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:50.785672 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b14fc0048141e280970f10a194f0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.786500 3409 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Bootstrap starting.
I20260621 07:03:50.787333 3412 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Bootstrap starting.
I20260621 07:03:50.788267 3409 tablet_bootstrap.cc:654] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.788964 3412 tablet_bootstrap.cc:654] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.796873 3434 raft_consensus.cc:2229] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:03:50.798300 3434 raft_consensus.cc:2258] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:50.800402 3409 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: No bootstrap required, opened a new log
I20260621 07:03:50.800634 3412 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: No bootstrap required, opened a new log
I20260621 07:03:50.800619 3409 ts_tablet_manager.cc:1397] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20260621 07:03:50.800802 3412 ts_tablet_manager.cc:1397] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260621 07:03:50.800972 3434 ts_tablet_manager.cc:1905] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:50.801590 3409 raft_consensus.cc:348] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.801621 3412 raft_consensus.cc:348] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.801847 3412 raft_consensus.cc:374] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.801951 3412 raft_consensus.cc:729] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daba67ffee44545b332ba05af07fbd5, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.801975 3409 raft_consensus.cc:374] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.802155 3409 raft_consensus.cc:729] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babf607b935d487e84012f218b6489bd, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.802215 3412 consensus_queue.cc:260] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.802558 3409 consensus_queue.cc:260] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.802790 3412 ts_tablet_manager.cc:1428] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 07:03:50.806149 3409 ts_tablet_manager.cc:1428] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 07:03:50.808576 3435 ts_tablet_manager.cc:1918] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:03:50.808826 3435 log.cc:1196] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/5c810bc59dfb4428ad82db5435d805ec
I20260621 07:03:50.809512 3435 ts_tablet_manager.cc:1939] T 5c810bc59dfb4428ad82db5435d805ec P 0daba67ffee44545b332ba05af07fbd5: Deleting consensus metadata
I20260621 07:03:50.811380 3411 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Bootstrap starting.
I20260621 07:03:50.811695 2892 catalog_manager.cc:4841] TS 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581): tablet 5c810bc59dfb4428ad82db5435d805ec (table test-table [id=d029fc34630a449abffd6bb2b0df3db2]) successfully deleted
I20260621 07:03:50.812566 3433 tablet_replica.cc:330] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: stopping tablet replica
I20260621 07:03:50.812923 3433 raft_consensus.cc:2229] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:50.813226 3411 tablet_bootstrap.cc:654] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.813309 3433 raft_consensus.cc:2258] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:50.815380 3433 ts_tablet_manager.cc:1905] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:50.816370 3434 ts_tablet_manager.cc:1918] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:03:50.816601 3434 log.cc:1196] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/5c810bc59dfb4428ad82db5435d805ec
I20260621 07:03:50.817310 3434 ts_tablet_manager.cc:1939] T 5c810bc59dfb4428ad82db5435d805ec P 4f8f8d0bea38465b8e35a63bc878f941: Deleting consensus metadata
I20260621 07:03:50.819078 2891 catalog_manager.cc:4841] TS 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193:42097): tablet 5c810bc59dfb4428ad82db5435d805ec (table test-table [id=d029fc34630a449abffd6bb2b0df3db2]) successfully deleted
I20260621 07:03:50.821631 3433 ts_tablet_manager.cc:1918] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:03:50.821832 3433 log.cc:1196] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/5c810bc59dfb4428ad82db5435d805ec
I20260621 07:03:50.822615 3433 ts_tablet_manager.cc:1939] T 5c810bc59dfb4428ad82db5435d805ec P babf607b935d487e84012f218b6489bd: Deleting consensus metadata
I20260621 07:03:50.825870 2889 catalog_manager.cc:4841] TS babf607b935d487e84012f218b6489bd (127.1.188.195:41911): tablet 5c810bc59dfb4428ad82db5435d805ec (table test-table [id=d029fc34630a449abffd6bb2b0df3db2]) successfully deleted
I20260621 07:03:50.826452 3411 tablet_bootstrap.cc:492] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: No bootstrap required, opened a new log
I20260621 07:03:50.826644 3411 ts_tablet_manager.cc:1397] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20260621 07:03:50.827387 3411 raft_consensus.cc:348] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.827698 3411 raft_consensus.cc:374] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.827814 3411 raft_consensus.cc:729] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f8d0bea38465b8e35a63bc878f941, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.828085 3411 consensus_queue.cc:260] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.828722 3411 ts_tablet_manager.cc:1428] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:03:50.882473 3430 raft_consensus.cc:482] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:50.882769 3430 raft_consensus.cc:504] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.883900 3430 leader_election.cc:288] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581), babf607b935d487e84012f218b6489bd (127.1.188.195:41911)
I20260621 07:03:50.884446 3206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b14fc0048141e280970f10a194f0ed" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daba67ffee44545b332ba05af07fbd5" is_pre_election: true
I20260621 07:03:50.884473 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b14fc0048141e280970f10a194f0ed" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd" is_pre_election: true
I20260621 07:03:50.884737 3340 raft_consensus.cc:2454] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 0.
I20260621 07:03:50.884732 3206 raft_consensus.cc:2454] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 0.
I20260621 07:03:50.885229 2992 leader_election.cc:302] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f8d0bea38465b8e35a63bc878f941, babf607b935d487e84012f218b6489bd; no voters:
I20260621 07:03:50.885509 3430 raft_consensus.cc:2781] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:50.885650 3430 raft_consensus.cc:482] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:50.885759 3430 raft_consensus.cc:3037] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.888995 3430 raft_consensus.cc:504] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.890000 3430 leader_election.cc:288] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 election: Requested vote from peers 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581), babf607b935d487e84012f218b6489bd (127.1.188.195:41911)
I20260621 07:03:50.890070 3206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b14fc0048141e280970f10a194f0ed" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daba67ffee44545b332ba05af07fbd5"
I20260621 07:03:50.890272 3206 raft_consensus.cc:3037] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.893127 3206 raft_consensus.cc:2454] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 1.
I20260621 07:03:50.893504 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b14fc0048141e280970f10a194f0ed" candidate_uuid: "4f8f8d0bea38465b8e35a63bc878f941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd"
I20260621 07:03:50.893602 2995 leader_election.cc:302] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daba67ffee44545b332ba05af07fbd5, 4f8f8d0bea38465b8e35a63bc878f941; no voters:
I20260621 07:03:50.893739 3340 raft_consensus.cc:3037] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:50.894677 3430 raft_consensus.cc:2781] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:50.894907 3430 raft_consensus.cc:686] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 LEADER]: Becoming Leader. State: Replica: 4f8f8d0bea38465b8e35a63bc878f941, State: Running, Role: LEADER
I20260621 07:03:50.895187 3430 consensus_queue.cc:237] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.896629 3340 raft_consensus.cc:2454] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f8f8d0bea38465b8e35a63bc878f941 in term 1.
I20260621 07:03:50.898981 2904 catalog_manager.cc:5495] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193). New cstate: current_term: 1 leader_uuid: "4f8f8d0bea38465b8e35a63bc878f941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:03:50 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025428, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:50.963999 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.950174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.1.188.254:36413)
0621 07:03:50.950387 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:50.950395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:50.951017 (+ 622us) server_negotiation.cc:407] Connection header received
0621 07:03:50.951244 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:50.951250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:50.951320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:50.951451 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:50.952664 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.953421 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.954670 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.955118 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.955244 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.958745 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:50.958750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:50.958753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:50.958780 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:50.961320 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.961981 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.961988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.961991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.962114 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:50.962473 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:50.962480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:50.962483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:50.962772 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:50.962924 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:50.963347 (+ 423us) server_negotiation.cc:299] Negotiation successful
0621 07:03:50.963703 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":69}
I20260621 07:03:50.964035 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:50.950274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:36413 (local address 127.0.0.1:58136)
0621 07:03:50.950894 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:50.950913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:50.951118 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:50.951493 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:50.951499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:50.952219 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:50.952492 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.952502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.953558 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.953565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.954530 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:50.954541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:50.955304 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:50.955310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:50.955582 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:50.956290 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:50.956308 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:50.958551 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:50.961453 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.961467 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.961475 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.961866 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.962218 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:50.962225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:50.962228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:50.962372 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:50.962905 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:50.962913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:50.963072 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 07:03:50.963251 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 07:03:50.963575 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":357,"threads_started":1}
I20260621 07:03:50.969470 2904 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58136:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:03:50.970487 2904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:50.978363 3042 tablet_service.cc:1467] Processing CreateTablet for tablet c9880b6a4b1c4f288bd424dc1bcae933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbf998c1930542e38f34cf5dfc144e2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:50.978434 3181 tablet_service.cc:1467] Processing CreateTablet for tablet c9880b6a4b1c4f288bd424dc1bcae933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbf998c1930542e38f34cf5dfc144e2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:50.978434 3320 tablet_service.cc:1467] Processing CreateTablet for tablet c9880b6a4b1c4f288bd424dc1bcae933 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cbf998c1930542e38f34cf5dfc144e2b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:50.978991 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9880b6a4b1c4f288bd424dc1bcae933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.978986 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9880b6a4b1c4f288bd424dc1bcae933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.978986 3042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9880b6a4b1c4f288bd424dc1bcae933. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:50.986387 3409 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: Bootstrap starting.
I20260621 07:03:50.986490 3411 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: Bootstrap starting.
I20260621 07:03:50.987914 3409 tablet_bootstrap.cc:654] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.988171 3411 tablet_bootstrap.cc:654] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.989161 3412 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: Bootstrap starting.
I20260621 07:03:50.990300 3409 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: No bootstrap required, opened a new log
I20260621 07:03:50.990446 3409 ts_tablet_manager.cc:1397] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:03:50.990851 3412 tablet_bootstrap.cc:654] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:50.991281 3409 raft_consensus.cc:348] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.991578 3409 raft_consensus.cc:374] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.991652 3409 raft_consensus.cc:729] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babf607b935d487e84012f218b6489bd, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.991938 3409 consensus_queue.cc:260] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.992547 3409 ts_tablet_manager.cc:1428] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:03:50.992415 3413 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:50.992795 3413 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:50.995729 3411 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: No bootstrap required, opened a new log
I20260621 07:03:50.995914 3411 ts_tablet_manager.cc:1397] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260621 07:03:50.997211 3411 raft_consensus.cc:348] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.997633 3411 raft_consensus.cc:374] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:50.997742 3411 raft_consensus.cc:729] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f8d0bea38465b8e35a63bc878f941, State: Initialized, Role: FOLLOWER
I20260621 07:03:50.998068 3411 consensus_queue.cc:260] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:50.998687 3411 ts_tablet_manager.cc:1428] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:03:51.000862 3430 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.001160 3430 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.001448 3412 tablet_bootstrap.cc:492] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: No bootstrap required, opened a new log
I20260621 07:03:51.001621 3412 ts_tablet_manager.cc:1397] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260621 07:03:51.002377 3412 raft_consensus.cc:348] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
W20260621 07:03:51.002667 3252 tablet.cc:2369] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:03:51.002713 3412 raft_consensus.cc:374] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:51.002810 3412 raft_consensus.cc:729] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daba67ffee44545b332ba05af07fbd5, State: Initialized, Role: FOLLOWER
I20260621 07:03:51.003095 3412 consensus_queue.cc:260] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:51.003621 3416 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.003983 3416 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.005088 3412 ts_tablet_manager.cc:1428] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20260621 07:03:51.010689 3391 tablet.cc:2369] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:03:51.085069 3416 raft_consensus.cc:482] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:51.085286 3416 raft_consensus.cc:504] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:51.087067 3416 leader_election.cc:288] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193:42097), babf607b935d487e84012f218b6489bd (127.1.188.195:41911)
I20260621 07:03:51.095628 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.087635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:58723 (local address 127.1.188.195:41911)
0621 07:03:51.087883 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.087893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.089302 (+ 1409us) server_negotiation.cc:407] Connection header received
0621 07:03:51.089617 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.089622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.089723 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.089805 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:51.090657 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.091595 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.093314 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.094491 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.094651 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.094731 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.095154 (+ 423us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.095300 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:03:51.095949 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.088108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:35113 (local address 127.1.188.193:42097)
0621 07:03:51.088341 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.088349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.089039 (+ 690us) server_negotiation.cc:407] Connection header received
0621 07:03:51.089353 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.089357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.089418 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.089520 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:51.090021 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.091013 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.093656 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.094900 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.095074 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.095151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.095490 (+ 339us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.095631 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:03:51.096022 3450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.087352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42097 (local address 127.1.188.194:35113)
0621 07:03:51.088909 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:51.088930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:51.089182 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:51.089537 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:51.089541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:51.089574 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:51.089869 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.089877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.091183 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.091189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.093497 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.093511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.095129 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.095136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.095305 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.095327 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:03:51.095463 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 07:03:51.095592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"spinlock_wait_cycles":18304,"thread_start_us":995,"threads_started":1}
I20260621 07:03:51.096813 3449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.087496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:41911 (local address 127.1.188.194:58723)
0621 07:03:51.089185 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:51.089206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:51.089414 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:51.090010 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:51.090015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:51.090045 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:51.090456 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.090472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.091723 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.091729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.093159 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.093171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.094721 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.094728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.094936 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.094975 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:03:51.096338 (+ 1363us) client_negotiation.cc:240] Negotiation successful
0621 07:03:51.096485 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1559,"thread_start_us":1390,"threads_started":1}
I20260621 07:03:51.097178 3067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9880b6a4b1c4f288bd424dc1bcae933" candidate_uuid: "0daba67ffee44545b332ba05af07fbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f8d0bea38465b8e35a63bc878f941" is_pre_election: true
I20260621 07:03:51.097460 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9880b6a4b1c4f288bd424dc1bcae933" candidate_uuid: "0daba67ffee44545b332ba05af07fbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd" is_pre_election: true
I20260621 07:03:51.097749 3340 raft_consensus.cc:2454] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daba67ffee44545b332ba05af07fbd5 in term 0.
I20260621 07:03:51.097718 3067 raft_consensus.cc:2454] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daba67ffee44545b332ba05af07fbd5 in term 0.
I20260621 07:03:51.098325 3132 leader_election.cc:302] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daba67ffee44545b332ba05af07fbd5, babf607b935d487e84012f218b6489bd; no voters:
I20260621 07:03:51.099048 3416 raft_consensus.cc:2781] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:51.099191 3416 raft_consensus.cc:482] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:51.099305 3416 raft_consensus.cc:3037] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:51.101401 3416 raft_consensus.cc:504] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:51.102075 3416 leader_election.cc:288] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [CANDIDATE]: Term 1 election: Requested vote from peers 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193:42097), babf607b935d487e84012f218b6489bd (127.1.188.195:41911)
I20260621 07:03:51.102560 3067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9880b6a4b1c4f288bd424dc1bcae933" candidate_uuid: "0daba67ffee44545b332ba05af07fbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f8d0bea38465b8e35a63bc878f941"
I20260621 07:03:51.102638 3340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9880b6a4b1c4f288bd424dc1bcae933" candidate_uuid: "0daba67ffee44545b332ba05af07fbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babf607b935d487e84012f218b6489bd"
I20260621 07:03:51.102785 3067 raft_consensus.cc:3037] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:51.102854 3340 raft_consensus.cc:3037] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:51.105876 3340 raft_consensus.cc:2454] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daba67ffee44545b332ba05af07fbd5 in term 1.
I20260621 07:03:51.106246 3067 raft_consensus.cc:2454] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daba67ffee44545b332ba05af07fbd5 in term 1.
I20260621 07:03:51.106369 3132 leader_election.cc:302] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daba67ffee44545b332ba05af07fbd5, babf607b935d487e84012f218b6489bd; no voters:
I20260621 07:03:51.106803 3416 raft_consensus.cc:2781] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:51.107311 3416 raft_consensus.cc:686] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [term 1 LEADER]: Becoming Leader. State: Replica: 0daba67ffee44545b332ba05af07fbd5, State: Running, Role: LEADER
I20260621 07:03:51.107746 3416 consensus_queue.cc:237] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } }
I20260621 07:03:51.109691 3451 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: TxnStatusTablet state changed. Reason: New leader 0daba67ffee44545b332ba05af07fbd5. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.110121 3451 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:51.110936 3453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:03:51.111172 3453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:03:51.111680 2904 catalog_manager.cc:5495] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194). New cstate: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } health_report { overall_health: UNKNOWN } } }
I20260621 07:03:51.140298 3445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.132015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:38581 (local address 127.0.0.1:48718)
0621 07:03:51.132388 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:51.132409 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:51.132579 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:51.133284 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:51.133289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:51.133302 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:51.133492 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.133504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.135255 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.135260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.136953 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.136973 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.138113 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.138120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.138335 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.138417 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.139653 (+ 1236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.139675 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:03:51.139761 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 07:03:51.139953 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260621 07:03:51.141434 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.132085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.1.188.194:38581)
0621 07:03:51.132271 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.132278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.132495 (+ 217us) server_negotiation.cc:407] Connection header received
0621 07:03:51.132804 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.132809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.132910 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.133081 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:51.133673 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.135006 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.137146 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.137809 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.137971 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.138668 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.139450 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.139596 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.140954 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.141052 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:03:51.146215 3340 raft_consensus.cc:1261] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Refusing update from remote peer 0daba67ffee44545b332ba05af07fbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:51.146595 3067 raft_consensus.cc:1261] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Refusing update from remote peer 0daba67ffee44545b332ba05af07fbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:51.147202 3416 consensus_queue.cc:1035] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:51.147938 3451 consensus_queue.cc:1035] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:51.155169 3413 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: TxnStatusTablet state changed. Reason: New leader 0daba67ffee44545b332ba05af07fbd5. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.155668 3413 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.169126 3430 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: TxnStatusTablet state changed. Reason: New leader 0daba67ffee44545b332ba05af07fbd5. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.169480 3430 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.170684 3416 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.171012 3416 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:51.172521 3424 mvcc.cc:205] Tried to move back new op lower bound from 7299176165967380480 to 7299176165820882944. Current Snapshot: MvccSnapshot[applied={T|T < 7299176165967380480}]
I20260621 07:03:51.172847 3413 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.173059 3413 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P babf607b935d487e84012f218b6489bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.179857 3430 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.180141 3430 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 4f8f8d0bea38465b8e35a63bc878f941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:51.182345 3451 tablet_replica.cc:439] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0daba67ffee44545b332ba05af07fbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f8f8d0bea38465b8e35a63bc878f941" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42097 } } peers { permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 } } peers { permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 } } }
I20260621 07:03:51.182778 3451 tablet_replica.cc:442] T c9880b6a4b1c4f288bd424dc1bcae933 P 0daba67ffee44545b332ba05af07fbd5: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:51.197152 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.189124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60993 (local address 127.1.188.194:38581)
0621 07:03:51.189330 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.189340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.192086 (+ 2746us) server_negotiation.cc:407] Connection header received
0621 07:03:51.192324 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.192329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.192410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.192510 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:51.192994 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.193732 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.195232 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.195766 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.195908 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.196370 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.196486 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.196586 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.196745 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.196827 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:03:51.197263 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.189123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:38581 (local address 127.1.188.193:60993)
0621 07:03:51.191969 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:51.191986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:51.192171 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:51.192530 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:51.192535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:51.192548 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:51.192803 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.192813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.193886 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.193891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.195078 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.195091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.195958 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.195966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.196110 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.196170 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.196589 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.196606 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 07:03:51.196713 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:03:51.196861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2658,"thread_start_us":2509,"threads_started":1}
I20260621 07:03:51.211215 3340 raft_consensus.cc:1261] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Refusing update from remote peer 4f8f8d0bea38465b8e35a63bc878f941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:51.212080 3430 consensus_queue.cc:1035] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babf607b935d487e84012f218b6489bd" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:51.217422 3206 raft_consensus.cc:1261] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Refusing update from remote peer 4f8f8d0bea38465b8e35a63bc878f941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:51.218021 3430 consensus_queue.cc:1035] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daba67ffee44545b332ba05af07fbd5" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:51.221001 3423 mvcc.cc:205] Tried to move back new op lower bound from 7299176166235340800 to 7299176164949176320. Current Snapshot: MvccSnapshot[applied={T|T < 7299176166235340800}]
I20260621 07:03:51.263602 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.256738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:38317 (local address 127.1.188.193:42097)
0621 07:03:51.256927 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.256934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.257059 (+ 125us) server_negotiation.cc:407] Connection header received
0621 07:03:51.257395 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.257401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.257487 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.257549 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:51.258777 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.259765 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.261391 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.262015 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.262111 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.262797 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.262913 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.262963 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.263154 (+ 191us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.263230 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:03:51.264000 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.256267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42097 (local address 127.1.188.194:38317)
0621 07:03:51.256959 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:51.256980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:51.257191 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:51.257745 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:51.257750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:51.257763 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:51.258573 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.258587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.259904 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.259907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.261224 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:51.261238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.262322 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.262329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:51.262510 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.262593 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:51.263024 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:51.263031 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:03:51.263120 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 07:03:51.263261 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":1838,"threads_started":1}
I20260621 07:03:51.304677 2903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:51.304952 2903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:51.307910 2903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bafb1152b45450f97a9cd186201c2eb: Sending DeleteTablet for 3 replicas of tablet 86b14fc0048141e280970f10a194f0ed
I20260621 07:03:51.308722 3181 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b14fc0048141e280970f10a194f0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:51 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:48714
I20260621 07:03:51.308738 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b14fc0048141e280970f10a194f0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:51 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58934
I20260621 07:03:51.309010 3435 tablet_replica.cc:330] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: stopping tablet replica
I20260621 07:03:51.309079 3433 tablet_replica.cc:330] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: stopping tablet replica
I20260621 07:03:51.309231 3435 raft_consensus.cc:2229] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:51.309273 3433 raft_consensus.cc:2229] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:51.309279 3042 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b14fc0048141e280970f10a194f0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:51 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59560
I20260621 07:03:51.309437 3433 raft_consensus.cc:2258] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:51.309435 3435 raft_consensus.cc:2258] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:51.309702 3434 tablet_replica.cc:330] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: stopping tablet replica
I20260621 07:03:51.309891 3434 raft_consensus.cc:2229] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:03:51.310191 3434 raft_consensus.cc:2258] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:51.311021 3435 ts_tablet_manager.cc:1905] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:51.311148 3433 ts_tablet_manager.cc:1905] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:51.316359 3434 ts_tablet_manager.cc:1905] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:51.320801 3435 ts_tablet_manager.cc:1918] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:51.320992 3435 log.cc:1196] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/86b14fc0048141e280970f10a194f0ed
I20260621 07:03:51.321566 3435 ts_tablet_manager.cc:1939] T 86b14fc0048141e280970f10a194f0ed P 0daba67ffee44545b332ba05af07fbd5: Deleting consensus metadata
I20260621 07:03:51.323035 2892 catalog_manager.cc:4841] TS 0daba67ffee44545b332ba05af07fbd5 (127.1.188.194:38581): tablet 86b14fc0048141e280970f10a194f0ed (table test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]) successfully deleted
I20260621 07:03:51.325846 3433 ts_tablet_manager.cc:1918] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:51.326002 3433 log.cc:1196] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/86b14fc0048141e280970f10a194f0ed
I20260621 07:03:51.326416 3433 ts_tablet_manager.cc:1939] T 86b14fc0048141e280970f10a194f0ed P babf607b935d487e84012f218b6489bd: Deleting consensus metadata
I20260621 07:03:51.327543 3434 ts_tablet_manager.cc:1918] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:51.327577 2889 catalog_manager.cc:4841] TS babf607b935d487e84012f218b6489bd (127.1.188.195:41911): tablet 86b14fc0048141e280970f10a194f0ed (table test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]) successfully deleted
I20260621 07:03:51.327710 3434 log.cc:1196] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/86b14fc0048141e280970f10a194f0ed
I20260621 07:03:51.328109 3434 ts_tablet_manager.cc:1939] T 86b14fc0048141e280970f10a194f0ed P 4f8f8d0bea38465b8e35a63bc878f941: Deleting consensus metadata
I20260621 07:03:51.329052 2891 catalog_manager.cc:4841] TS 4f8f8d0bea38465b8e35a63bc878f941 (127.1.188.193:42097): tablet 86b14fc0048141e280970f10a194f0ed (table test-table [id=19d3a45a35f342a79d7f2701eac4bdf5]) successfully deleted
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:03:51.343276 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.320163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.1.188.193:42097)
0621 07:03:51.320361 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.320367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.324016 (+ 3649us) server_negotiation.cc:407] Connection header received
0621 07:03:51.324260 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.324273 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.324363 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.324472 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:51.325796 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.326687 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.328342 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.328781 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.328911 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.337196 (+ 8285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:51.337211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:51.337228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:51.337284 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:51.340277 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.341100 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.341108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.341110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.341242 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.341692 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.341696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.341698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.342127 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:51.342359 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.342737 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.342920 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 07:03:51.344095 3107 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59584
I20260621 07:03:51.361109 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.347223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.1.188.254:36413)
0621 07:03:51.347468 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.347473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.347495 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:03:51.347690 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.347694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.347754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.347844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:51.349630 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.350540 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.352427 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.352827 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.352918 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.355818 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:51.355823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:51.355826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:51.355857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:51.358520 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.359313 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.359317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.359319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.359494 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.359847 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.359850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.359852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.360098 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:51.360312 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.360640 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.360807 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 07:03:51.361745 2903 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58148
I20260621 07:03:51.363762 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2976
I20260621 07:03:51.378732 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.365174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.1.188.193:42097)
0621 07:03:51.365379 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.365385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.365542 (+ 157us) server_negotiation.cc:407] Connection header received
0621 07:03:51.365664 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.365669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.365748 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.365852 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:51.367472 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.368589 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.369754 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.370423 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.370550 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.372745 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:51.372752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:51.372756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:51.372795 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:51.376105 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.376912 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.376918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.376922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.377062 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.377404 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.377410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.377414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.377762 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:51.377992 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.378181 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.378397 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260621 07:03:51.379427 3107 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59590
W20260621 07:03:51.379988 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:03:51.380280 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2976
I20260621 07:03:51.402465 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3116
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:03:51.419445 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.403227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.1.188.194:38581)
0621 07:03:51.403506 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.403512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.403535 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:03:51.403685 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.403690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.403770 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.403883 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:51.405036 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.406167 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.407479 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.408201 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.408359 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.413018 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:51.413029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:51.413042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:51.413086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:51.416551 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.417374 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.417380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.417382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.417514 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.417918 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.417921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.417923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.418299 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:51.418545 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.418902 (+ 357us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.419080 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 07:03:51.420274 3246 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48720
W20260621 07:03:51.420965 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3116: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:03:51.421177 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3116
I20260621 07:03:51.442619 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3255
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[2860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025430, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:03:51.458562 3404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:51.443387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.1.188.195:41911)
0621 07:03:51.443600 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:51.443605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:51.443629 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:03:51.443860 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:51.443865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:51.443935 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:51.444026 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:51.445307 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.446044 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.447237 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:51.447934 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:51.448033 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:51.452316 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:51.452334 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:51.452351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:51.452400 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:51.455350 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.456429 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.456439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.456441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.456563 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:51.456915 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:51.456918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:51.456920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:51.457358 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:51.457581 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:51.457995 (+ 414us) server_negotiation.cc:299] Negotiation successful
0621 07:03:51.458233 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 07:03:51.459232 3385 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58948
W20260621 07:03:51.459896 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3255: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:03:51.460062 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3255
I20260621 07:03:51.476601 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 2873
I20260621 07:03:51.477895 2945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:03:51.592264 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 2873
2026-06-21T07:03:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4420 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[3476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[3476](info): set up 2 sockets
Jun 21 07:03:51 dist-test-slave-hlq1 krb5kdc[3476](info): commencing operation
krb5kdc: starting...
W20260621 07:03:52.679754 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:03:52 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:03:52.865152 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35847
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:53.182890 3489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:53.183182 3489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:53.183387 3489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:53.190667 3489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:03:53.190760 3489 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:03:53.190850 3489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:53.190959 3489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:53.191025 3489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:03:53.191155 3489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:03:53.201159 3489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:35847
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:53.202488 3489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:53.203975 3489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:53.215775 3494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:53.215807 3495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:53.215989 3497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:53.216717 3489 server_base.cc:1034] running on GCE node
I20260621 07:03:53.217536 3489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:53.218955 3489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:53.220216 3489 hybrid_clock.cc:648] HybridClock initialized: now 1782025433220149 us; error 37 us; skew 500 ppm
I20260621 07:03:53.220559 3489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:53 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025433, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:53.223757 3489 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:03:53.225286 3489 webserver.cc:460] Webserver started at http://127.1.188.254:43743/ using document root <none> and password file <none>
I20260621 07:03:53.225982 3489 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:53.226111 3489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:53.226429 3489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:53.229424 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "36bb5eff83f04f77ac30c336ac583d3b"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "03eefae37d80761238e522d29801504b"
server_key_iv: "1f752f1dab4fc093c6b280119d6bc55e"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.230183 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "36bb5eff83f04f77ac30c336ac583d3b"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "03eefae37d80761238e522d29801504b"
server_key_iv: "1f752f1dab4fc093c6b280119d6bc55e"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.235378 3489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 07:03:53.238919 3504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:53.240329 3489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:03:53.240516 3489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "36bb5eff83f04f77ac30c336ac583d3b"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "03eefae37d80761238e522d29801504b"
server_key_iv: "1f752f1dab4fc093c6b280119d6bc55e"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.240744 3489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:53.266242 3489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:53.270378 3489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:53.270650 3489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:53.291882 3489 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:35847
I20260621 07:03:53.291899 3565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:35847 every 8 connection(s)
I20260621 07:03:53.293223 3489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:03:53.297052 3566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:53.302845 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3489
I20260621 07:03:53.303058 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:03:53.303946 1779 external_mini_cluster.cc:1428] Setting key 29c4d0c957aa5c3812cf08f8b22b7a61
I20260621 07:03:53.308792 3566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Bootstrap starting.
I20260621 07:03:53.312484 3566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:53.313740 3566 log.cc:824] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:53.316917 3566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: No bootstrap required, opened a new log
Jun 21 07:03:53 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:53.325269 3566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } }
I20260621 07:03:53.325680 3566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:53.325814 3566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bb5eff83f04f77ac30c336ac583d3b, State: Initialized, Role: FOLLOWER
I20260621 07:03:53.326562 3566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } }
I20260621 07:03:53.326836 3566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:03:53.326977 3566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:03:53.327153 3566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:53.328086 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:53.306140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.1.188.254:35847)
0621 07:03:53.306995 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:53.307006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:53.307047 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:03:53.308749 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:53.308772 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:53.309208 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:53.309511 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:53.311359 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.312787 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.314604 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.315479 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.315608 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:53.320329 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:53.320337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:53.320351 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:53.320386 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:53.323613 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.324319 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.324326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.324332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.324494 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.324864 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.324869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.324872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.325375 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:53.325689 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:53.326156 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 07:03:53.326608 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":422,"threads_started":1}
I20260621 07:03:53.329820 3566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } }
I20260621 07:03:53.330338 3566 leader_election.cc:302] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bb5eff83f04f77ac30c336ac583d3b; no voters:
I20260621 07:03:53.330971 3566 leader_election.cc:288] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:03:53.331097 3571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:53.331861 3571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [term 1 LEADER]: Becoming Leader. State: Replica: 36bb5eff83f04f77ac30c336ac583d3b, State: Running, Role: LEADER
I20260621 07:03:53.332525 3571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } }
I20260621 07:03:53.333549 3566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:03:53.335597 3572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36bb5eff83f04f77ac30c336ac583d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } } }
I20260621 07:03:53.336077 3572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:53.339730 3573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bb5eff83f04f77ac30c336ac583d3b. Latest consensus state: current_term: 1 leader_uuid: "36bb5eff83f04f77ac30c336ac583d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bb5eff83f04f77ac30c336ac583d3b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35847 } } }
I20260621 07:03:53.340131 3577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:03:53.340306 3573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b [sys.catalog]: This master's current role is: LEADER
I20260621 07:03:53.345937 3577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:03:53.355262 3577 catalog_manager.cc:1347] Generated new cluster ID: 6f340e5801454e1e854d57152f3470b8
I20260621 07:03:53.355415 3577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:03:53.376081 3577 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:03:53.378072 3577 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:03:53.386875 3577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Generated new TSK 0
I20260621 07:03:53.387928 3577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:03:53.439715 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35847
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:53.773305 3592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:53.773715 3592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:53.773967 3592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:53.781766 3592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:53.781921 3592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:53.782070 3592 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:03:53.782158 3592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:53.782265 3592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:03:53.792243 3592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:53.793874 3592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:53.795383 3592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:53.808177 3597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:53.808283 3600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:53.808176 3598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:53.809312 3592 server_base.cc:1034] running on GCE node
I20260621 07:03:53.809899 3592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:53.810817 3592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:53.812177 3592 hybrid_clock.cc:648] HybridClock initialized: now 1782025433811833 us; error 330 us; skew 500 ppm
I20260621 07:03:53.812495 3592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:53 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025433, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:53.815728 3592 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:03:53.817132 3592 webserver.cc:460] Webserver started at http://127.1.188.193:40321/ using document root <none> and password file <none>
I20260621 07:03:53.817806 3592 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:53.817934 3592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:53.818245 3592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:53.821188 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "fa1225def67f448b8dd4bfa3378c3aba"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "26136e453e1f191b60941fa0c122cbaf"
server_key_iv: "eb47f33943539084c8bee58ee0e54b9a"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.821931 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "fa1225def67f448b8dd4bfa3378c3aba"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "26136e453e1f191b60941fa0c122cbaf"
server_key_iv: "eb47f33943539084c8bee58ee0e54b9a"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.826961 3592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:03:53.830353 3607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:53.831775 3592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:03:53.831943 3592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "fa1225def67f448b8dd4bfa3378c3aba"
format_stamp: "Formatted at 2026-06-21 07:03:53 on dist-test-slave-hlq1"
server_key: "26136e453e1f191b60941fa0c122cbaf"
server_key_iv: "eb47f33943539084c8bee58ee0e54b9a"
server_key_version: "encryptionkey@0"
I20260621 07:03:53.832144 3592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:53.864562 3592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:53.868805 3592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:53.869132 3592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:53.871876 3592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:53.872025 3592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:53.872155 3592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:53.872251 3592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:53 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025433, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:53.897076 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:53.872560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:42677 (local address 127.1.188.254:35847)
0621 07:03:53.873182 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:53.873188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:53.874349 (+ 1161us) server_negotiation.cc:407] Connection header received
0621 07:03:53.877398 (+ 3049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:53.877407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:53.877499 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:53.877641 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:53.880041 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.881120 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.883614 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.884413 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.884571 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:53.889365 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:53.889374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:53.889379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:53.889426 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:53.892973 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.894035 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.894044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.894049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.894231 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.894887 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.894893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.894897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.895203 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:53.895386 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:53.896483 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0621 07:03:53.896706 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":366,"threads_started":1}
I20260621 07:03:53.898159 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:53.873260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.193:42677)
0621 07:03:53.874149 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:53.874207 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:53.876979 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:53.877961 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:53.877973 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:53.878801 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:53.879810 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:53.879857 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.881281 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.881289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:53.883405 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:53.883463 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.884726 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.884736 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:53.885060 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:53.886024 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:53.886058 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:53.889048 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:53.893108 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:53.893118 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:53.893135 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:53.893874 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:53.894385 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:53.894393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:53.894403 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:53.894718 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:53.895488 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:53.895497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:53.895994 (+ 497us) client_negotiation.cc:769] Sending connection context
0621 07:03:53.896387 (+ 393us) client_negotiation.cc:240] Negotiation successful
0621 07:03:53.896957 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":374,"threads_started":1}
I20260621 07:03:53.919734 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:53.904066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:35969 (local address 127.1.188.254:35847)
0621 07:03:53.904293 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:53.904301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:53.905121 (+ 820us) server_negotiation.cc:407] Connection header received
0621 07:03:53.905403 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:53.905407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:53.905473 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:53.905554 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:53.906972 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.907741 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.910506 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.911093 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.911242 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:53.913894 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:53.913899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:53.913903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:53.913933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:53.916500 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.917454 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.917458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.917460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.917577 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:53.918083 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:53.918106 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:53.918112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:53.918523 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:53.918711 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:53.919264 (+ 553us) server_negotiation.cc:299] Negotiation successful
0621 07:03:53.919498 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:03:53.920364 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:53.904305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.193:35969)
0621 07:03:53.904994 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:53.905020 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:53.905252 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:53.905614 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:53.905621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:53.906334 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:53.906798 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:53.906823 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.907885 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.907893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:53.910338 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:53.910360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:53.911326 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:53.911336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:53.911590 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:53.912300 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:53.912330 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:53.913684 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:53.916636 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:53.916644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:53.916651 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:53.917334 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:53.917693 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:53.917701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:53.917712 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:53.917894 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:53.918783 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:53.918787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:53.918999 (+ 212us) client_negotiation.cc:769] Sending connection context
0621 07:03:53.919168 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 07:03:53.919586 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":309,"threads_started":1}
I20260621 07:03:53.929293 3592 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:43689
I20260621 07:03:53.929322 3727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:43689 every 8 connection(s)
I20260621 07:03:53.930577 3592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:03:53.932623 3728 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35847
I20260621 07:03:53.932816 3728 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:53.933451 3728 heartbeater.cc:511] Master 127.1.188.254:35847 requested a full tablet report, sending...
I20260621 07:03:53.935640 3521 ts_manager.cc:194] Registered new tserver with Master: fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689)
I20260621 07:03:53.937204 3521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:42677
I20260621 07:03:53.940598 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3592
I20260621 07:03:53.940799 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:03:53.941483 1779 external_mini_cluster.cc:1428] Setting key 0c39446f143533314abe358aeb08e185
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:03:53.978391 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35847
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:54.294413 3732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:54.294734 3732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:54.294922 3732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:54.302492 3732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:54.302639 3732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:54.302798 3732 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:03:54.302886 3732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:54.303023 3732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:03:54.313040 3732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:54.314344 3732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:54.315721 3732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:54.328640 3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:54.328719 3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:54.328715 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:54.329984 3732 server_base.cc:1034] running on GCE node
I20260621 07:03:54.330528 3732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:54.331450 3732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:54.332651 3732 hybrid_clock.cc:648] HybridClock initialized: now 1782025434332600 us; error 47 us; skew 500 ppm
I20260621 07:03:54.332921 3732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:54.336285 3732 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:03:54.337875 3732 webserver.cc:460] Webserver started at http://127.1.188.194:45919/ using document root <none> and password file <none>
I20260621 07:03:54.338564 3732 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:54.338698 3732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:54.339026 3732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:54.342183 3732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "abc28f76126145baa703162ff44c54db"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "5506f7dbaa3a155379813013d0323d0e"
server_key_iv: "3572ccb291a188d5e8ffea6270a3f965"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.342926 3732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "abc28f76126145baa703162ff44c54db"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "5506f7dbaa3a155379813013d0323d0e"
server_key_iv: "3572ccb291a188d5e8ffea6270a3f965"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.348454 3732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 07:03:54.351907 3747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:54.353204 3732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:03:54.353380 3732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "abc28f76126145baa703162ff44c54db"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "5506f7dbaa3a155379813013d0323d0e"
server_key_iv: "3572ccb291a188d5e8ffea6270a3f965"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.353577 3732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:54.370402 3732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:54.374904 3732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:54.375221 3732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:54.378062 3732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:54.378233 3732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:54.378369 3732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:54.378469 3732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:54.407492 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.378887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:43187 (local address 127.1.188.254:35847)
0621 07:03:54.379062 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:54.379069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:54.380643 (+ 1574us) server_negotiation.cc:407] Connection header received
0621 07:03:54.383889 (+ 3246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:54.383898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:54.384019 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:54.384101 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:54.387135 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.388570 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.392016 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.392705 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.392867 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.399000 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:54.399011 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:54.399019 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:54.399071 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:54.403346 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.404798 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.404806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.404811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.404982 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.405604 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.405609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.405612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.405902 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:54.406074 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:54.406863 (+ 789us) server_negotiation.cc:299] Negotiation successful
0621 07:03:54.407092 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:03:54.408987 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.379528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.194:43187)
0621 07:03:54.380439 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:54.380502 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:54.383448 (+ 2946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:54.384778 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:54.384796 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:54.385794 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:54.386887 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.386916 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.388825 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.388832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.391797 (+ 2965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.391817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.392999 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.393008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.393282 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.394172 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:54.394223 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:54.398594 (+ 4371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:54.403664 (+ 5070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.403677 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.403703 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.404604 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.405177 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.405201 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.405209 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.405456 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.406043 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:54.406051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:54.406494 (+ 443us) client_negotiation.cc:769] Sending connection context
0621 07:03:54.406771 (+ 277us) client_negotiation.cc:240] Negotiation successful
0621 07:03:54.407350 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":343,"threads_started":1}
I20260621 07:03:54.423517 3732 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:39863
I20260621 07:03:54.423594 3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:39863 every 8 connection(s)
I20260621 07:03:54.424955 3732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:03:54.425971 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3732
I20260621 07:03:54.426147 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:03:54.426741 1779 external_mini_cluster.cc:1428] Setting key 7f2cddf180103f7953ab1a39fa181724
I20260621 07:03:54.429273 3867 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35847
I20260621 07:03:54.429644 3867 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:54.430670 3867 heartbeater.cc:511] Master 127.1.188.254:35847 requested a full tablet report, sending...
I20260621 07:03:54.432413 3520 ts_manager.cc:194] Registered new tserver with Master: abc28f76126145baa703162ff44c54db (127.1.188.194:39863)
I20260621 07:03:54.433610 3520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:43187
I20260621 07:03:54.434015 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.416736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:58539 (local address 127.1.188.254:35847)
0621 07:03:54.416970 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:54.416977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:54.417860 (+ 883us) server_negotiation.cc:407] Connection header received
0621 07:03:54.418100 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:54.418106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:54.418174 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:54.418319 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:54.419934 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.421457 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.422982 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.423446 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.423585 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.425481 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:54.425489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:54.425499 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:54.425529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:54.428825 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.431594 (+ 2769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.431602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.431609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.431774 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.432244 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.432251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.432257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.432736 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:54.432889 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:54.433368 (+ 479us) server_negotiation.cc:299] Negotiation successful
0621 07:03:54.433616 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:03:54.434031 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.416735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.194:58539)
0621 07:03:54.417703 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:54.417734 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:54.417935 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:54.418352 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:54.418357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:54.419144 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:54.419699 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.419725 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.421610 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.421616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.422827 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.422838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.423640 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.423646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.423860 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.424498 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:54.424514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:54.425300 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:54.428963 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.428975 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.428978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.429491 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.431912 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.431923 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.431929 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.432088 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.432878 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:54.432883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:54.433064 (+ 181us) client_negotiation.cc:769] Sending connection context
0621 07:03:54.433238 (+ 174us) client_negotiation.cc:240] Negotiation successful
0621 07:03:54.433566 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":580,"threads_started":1}
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:03:54.470252 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35847
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:03:54.792870 3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:03:54.793217 3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:03:54.793393 3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:03:54.800943 3871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:03:54.801095 3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:03:54.801244 3871 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 07:03:54.801329 3871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:03:54.801461 3871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:03:54.811512 3871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:35847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:03:54.812884 3871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:03:54.814167 3871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:03:54.827019 3876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:54.827041 3877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:03:54.826982 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:03:54.828076 3871 server_base.cc:1034] running on GCE node
I20260621 07:03:54.828739 3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:03:54.829708 3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:03:54.830929 3871 hybrid_clock.cc:648] HybridClock initialized: now 1782025434830870 us; error 47 us; skew 500 ppm
I20260621 07:03:54.831303 3871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:03:54.834784 3871 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:03:54.836553 3871 webserver.cc:460] Webserver started at http://127.1.188.195:44025/ using document root <none> and password file <none>
I20260621 07:03:54.837378 3871 fs_manager.cc:359] Metadata directory not provided
I20260621 07:03:54.837539 3871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:03:54.837884 3871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:03:54.841241 3871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "0bb4db43cd9747e5addd015a454dbb6a"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "f13626cff7a6f40349a291c10025dfe4"
server_key_iv: "8239fc5d0c6d419049d7508885bc211d"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.842061 3871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "0bb4db43cd9747e5addd015a454dbb6a"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "f13626cff7a6f40349a291c10025dfe4"
server_key_iv: "8239fc5d0c6d419049d7508885bc211d"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.847704 3871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 07:03:54.851061 3886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:54.852515 3871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:03:54.852700 3871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "0bb4db43cd9747e5addd015a454dbb6a"
format_stamp: "Formatted at 2026-06-21 07:03:54 on dist-test-slave-hlq1"
server_key: "f13626cff7a6f40349a291c10025dfe4"
server_key_iv: "8239fc5d0c6d419049d7508885bc211d"
server_key_version: "encryptionkey@0"
I20260621 07:03:54.852906 3871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:03:54.877511 3871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:03:54.881839 3871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:03:54.882169 3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:03:54.884986 3871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:03:54.885134 3871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:03:54.885315 3871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:03:54.885440 3871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:54.912585 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.885455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:44879 (local address 127.1.188.254:35847)
0621 07:03:54.885685 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:54.885695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:54.887217 (+ 1522us) server_negotiation.cc:407] Connection header received
0621 07:03:54.890389 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:54.890396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:54.890491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:54.890577 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:54.893748 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.894912 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.897971 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.898640 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.898818 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.904721 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:54.904732 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:54.904736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:54.904772 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:54.907928 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.909272 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.909284 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.909288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.909442 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.910047 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.910051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.910053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.910415 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:54.910551 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:54.911786 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0621 07:03:54.912090 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:03:54.914500 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.886147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.195:44879)
0621 07:03:54.886999 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:54.887057 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:54.889977 (+ 2920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:54.891276 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:54.891293 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:54.892411 (+ 1118us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:54.893494 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.893528 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.895115 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.895119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.897704 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.897727 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.899170 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.899176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.899517 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.900576 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:54.900623 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:54.904445 (+ 3822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:54.908119 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.908133 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.908234 (+ 101us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.909092 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.909567 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.909574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.909603 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.909911 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.910557 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:54.910569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:54.911214 (+ 645us) client_negotiation.cc:769] Sending connection context
0621 07:03:54.911622 (+ 408us) client_negotiation.cc:240] Negotiation successful
0621 07:03:54.912424 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":356,"threads_started":1}
I20260621 07:03:54.938669 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.921877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.1.188.195:51391)
0621 07:03:54.922936 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:54.922963 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:54.923239 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:54.923748 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:54.923753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:54.924401 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:54.924872 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.924891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.926218 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.926224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.927740 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:54.927757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.928789 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.928799 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:54.929051 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.929835 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:54.929865 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:54.931148 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:54.935876 (+ 4728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.935879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.935882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.936327 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.936829 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:54.936833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:54.936835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:54.936951 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:54.937648 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:54.937652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:54.937806 (+ 154us) client_negotiation.cc:769] Sending connection context
0621 07:03:54.937917 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:03:54.938267 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":713,"threads_started":1}
I20260621 07:03:54.939448 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.921932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:51391 (local address 127.1.188.254:35847)
0621 07:03:54.922162 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:54.922171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:54.923058 (+ 887us) server_negotiation.cc:407] Connection header received
0621 07:03:54.923536 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:54.923543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:54.923619 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:54.923726 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:54.925027 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.926069 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.927881 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.928574 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.928697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.931383 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:54.931394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:54.931424 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:54.931474 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:54.935722 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.936489 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.936497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.936504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.936702 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.937055 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.937064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.937070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.937519 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:54.937695 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:54.938086 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 07:03:54.938859 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:03:54.940147 3728 heartbeater.cc:503] Master 127.1.188.254:35847 was elected leader, sending a full tablet report...
I20260621 07:03:54.942549 3871 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:38035
I20260621 07:03:54.942574 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:38035 every 8 connection(s)
I20260621 07:03:54.944291 3871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:03:54.946465 4006 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35847
I20260621 07:03:54.946699 4006 heartbeater.cc:464] Registering TS with master...
I20260621 07:03:54.947347 4006 heartbeater.cc:511] Master 127.1.188.254:35847 requested a full tablet report, sending...
I20260621 07:03:54.948632 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 3871
I20260621 07:03:54.948814 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:03:54.948777 3520 ts_manager.cc:194] Registered new tserver with Master: 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:54.949266 1779 external_mini_cluster.cc:1428] Setting key db1c0ce5dd8cde296388bbeb2a0ff5ce
I20260621 07:03:54.949677 3520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:44879
I20260621 07:03:54.952344 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:03:54 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:54.997983 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:54.983778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.1.188.254:35847)
0621 07:03:54.983945 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:54.983950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:54.984321 (+ 371us) server_negotiation.cc:407] Connection header received
0621 07:03:54.984466 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:54.984471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:54.984536 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:54.984627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:54.985962 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.986735 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.988143 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:54.988560 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:54.988673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:54.992558 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:54.992564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:54.992567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:54.992607 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:54.995128 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.995914 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.995919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.995921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.996040 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:54.996477 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:54.996482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:54.996484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:54.996769 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:54.997037 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:54.997354 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 07:03:54.997591 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:03:55.002460 3520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:55.009464 3520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:55.045728 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.022621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:43689 (local address 127.0.0.1:43020)
0621 07:03:55.036847 (+ 14226us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.036864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.036976 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.037627 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.037631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.037651 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.038352 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.038369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.040270 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.040276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.042752 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.042767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.044709 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.044715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.044931 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.044989 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.045095 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.045304 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14115,"mutex_wait_us":256,"thread_start_us":19598,"threads_started":1}
I20260621 07:03:55.047138 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.023391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.1.188.193:43689)
0621 07:03:55.035430 (+ 12039us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.035439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.037080 (+ 1641us) server_negotiation.cc:407] Connection header received
0621 07:03:55.037189 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.037198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.037421 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.037723 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.038536 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.040110 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.042989 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.044482 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.046377 (+ 1895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.046601 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.046670 (+ 69us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.046841 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11300,"thread_start_us":11112,"threads_started":1}
I20260621 07:03:55.047263 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.022429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.0.0.1:59248)
0621 07:03:55.035514 (+ 13085us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.035573 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.035793 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.037411 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.037418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.037450 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.037906 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.037920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.041307 (+ 3387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.041313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.043474 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.043487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.046471 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.046477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.046647 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.046691 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.046753 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.046873 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12711,"mutex_wait_us":1,"thread_start_us":12542,"threads_started":1}
I20260621 07:03:55.048264 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.023074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.1.188.195:38035)
0621 07:03:55.036639 (+ 13565us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.036648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.036697 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:03:55.036801 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.036810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.037189 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.039161 (+ 1972us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.039495 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.041121 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.043679 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.046264 (+ 2585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.047499 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.047740 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.047803 (+ 63us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.047945 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13381,"thread_start_us":13194,"threads_started":1}
I20260621 07:03:55.050958 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 29422781001c4e2d9b7560ed730b1f04 (DEFAULT_TABLE table=test-table [id=654a234f951e4cbdb3596717dc850109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.051954 3936 tablet_service.cc:1467] Processing CreateTablet for tablet 29422781001c4e2d9b7560ed730b1f04 (DEFAULT_TABLE table=test-table [id=654a234f951e4cbdb3596717dc850109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.052539 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29422781001c4e2d9b7560ed730b1f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.053162 3936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29422781001c4e2d9b7560ed730b1f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.055743 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.036033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39863 (local address 127.0.0.1:42176)
0621 07:03:55.046076 (+ 10043us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.046096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.046205 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.047850 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.047858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.047897 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.048132 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.048146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.051355 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.051360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.053234 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.053248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.054944 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.054953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.055142 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.055176 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.055276 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.055445 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9947,"thread_start_us":12778,"threads_started":1}
I20260621 07:03:55.055821 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.036324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.1.188.194:39863)
0621 07:03:55.046908 (+ 10584us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.046921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.046956 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:03:55.047078 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.047089 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.047357 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.048357 (+ 1000us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.048579 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.049719 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.053421 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.054758 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.054905 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.055086 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.055336 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.055523 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10349,"thread_start_us":11202,"threads_started":1}
I20260621 07:03:55.059543 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 29422781001c4e2d9b7560ed730b1f04 (DEFAULT_TABLE table=test-table [id=654a234f951e4cbdb3596717dc850109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.060890 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29422781001c4e2d9b7560ed730b1f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.064687 4025 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Bootstrap starting.
I20260621 07:03:55.067054 4026 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Bootstrap starting.
I20260621 07:03:55.068459 4025 tablet_bootstrap.cc:654] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.069767 4025 log.cc:824] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:55.070780 4026 tablet_bootstrap.cc:654] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.072324 4026 log.cc:824] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:55.074895 4025 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: No bootstrap required, opened a new log
I20260621 07:03:55.075237 4025 ts_tablet_manager.cc:1397] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260621 07:03:55.075984 4029 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Bootstrap starting.
I20260621 07:03:55.077435 4026 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: No bootstrap required, opened a new log
I20260621 07:03:55.077708 4026 ts_tablet_manager.cc:1397] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260621 07:03:55.079523 4029 tablet_bootstrap.cc:654] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.080756 4029 log.cc:824] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Log is configured to *not* fsync() on all Append() calls
I20260621 07:03:55.084342 4029 tablet_bootstrap.cc:492] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: No bootstrap required, opened a new log
I20260621 07:03:55.084637 4029 ts_tablet_manager.cc:1397] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260621 07:03:55.086220 4026 raft_consensus.cc:348] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.086125 4025 raft_consensus.cc:348] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.086566 4025 raft_consensus.cc:374] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.086570 4026 raft_consensus.cc:374] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.086658 4026 raft_consensus.cc:729] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.086721 4025 raft_consensus.cc:729] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.087245 4026 consensus_queue.cc:260] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.087392 4025 consensus_queue.cc:260] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.088873 4026 ts_tablet_manager.cc:1428] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260621 07:03:55.090111 4006 heartbeater.cc:503] Master 127.1.188.254:35847 was elected leader, sending a full tablet report...
I20260621 07:03:55.090727 4025 ts_tablet_manager.cc:1428] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260621 07:03:55.094661 4029 raft_consensus.cc:348] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.095055 4029 raft_consensus.cc:374] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.095178 4029 raft_consensus.cc:729] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc28f76126145baa703162ff44c54db, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.095801 4029 consensus_queue.cc:260] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.096812 3867 heartbeater.cc:503] Master 127.1.188.254:35847 was elected leader, sending a full tablet report...
I20260621 07:03:55.097431 4029 ts_tablet_manager.cc:1428] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260621 07:03:55.182946 3868 tablet.cc:2369] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:03:55.183781 3729 tablet.cc:2369] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:03:55.196115 4007 tablet.cc:2369] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:03:55.233856 4031 raft_consensus.cc:482] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:55.234175 4031 raft_consensus.cc:504] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.236148 4031 leader_election.cc:288] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035), abc28f76126145baa703162ff44c54db (127.1.188.194:39863)
I20260621 07:03:55.243768 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.237716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:44095 (local address 127.1.188.194:39863)
0621 07:03:55.237944 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.237950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.238257 (+ 307us) server_negotiation.cc:407] Connection header received
0621 07:03:55.238442 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.238447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.238526 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.238635 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.239270 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.240124 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.242009 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.242781 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.242937 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.243023 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.243341 (+ 318us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.243496 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:03:55.243815 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.236555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:57723 (local address 127.1.188.195:38035)
0621 07:03:55.236780 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.236785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.237400 (+ 615us) server_negotiation.cc:407] Connection header received
0621 07:03:55.237898 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.237907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.237992 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.238102 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.238838 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.239852 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.242067 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.242823 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.243013 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.243087 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.243450 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.243581 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 07:03:55.243886 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.237503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39863 (local address 127.1.188.193:44095)
0621 07:03:55.238138 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.238158 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.238288 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.238653 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.238656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.238673 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.239064 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.239077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.240262 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.240266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.241856 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.241868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.242998 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.243007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.243170 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.243213 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.243284 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.243438 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":3,"thread_start_us":324,"threads_started":1}
I20260621 07:03:55.244012 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.236552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.1.188.193:57723)
0621 07:03:55.237260 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.237277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.237430 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.238171 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.238174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.238199 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.238673 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.238686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.239978 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.239983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.241899 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.241911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.243075 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.243088 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.243240 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.243282 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.243377 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.243513 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":140,"thread_start_us":393,"threads_started":1}
I20260621 07:03:55.245141 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29422781001c4e2d9b7560ed730b1f04" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a" is_pre_election: true
I20260621 07:03:55.245325 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29422781001c4e2d9b7560ed730b1f04" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db" is_pre_election: true
I20260621 07:03:55.245711 3956 raft_consensus.cc:2454] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 0.
I20260621 07:03:55.245816 3817 raft_consensus.cc:2454] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 0.
I20260621 07:03:55.246354 3608 leader_election.cc:302] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb4db43cd9747e5addd015a454dbb6a, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:55.246703 4031 raft_consensus.cc:2781] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:55.246861 4031 raft_consensus.cc:482] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:55.246984 4031 raft_consensus.cc:3037] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.249220 4031 raft_consensus.cc:504] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.249784 4031 leader_election.cc:288] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 election: Requested vote from peers 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035), abc28f76126145baa703162ff44c54db (127.1.188.194:39863)
I20260621 07:03:55.250260 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29422781001c4e2d9b7560ed730b1f04" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a"
I20260621 07:03:55.250309 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29422781001c4e2d9b7560ed730b1f04" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db"
I20260621 07:03:55.250488 3817 raft_consensus.cc:3037] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.250488 3956 raft_consensus.cc:3037] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.253670 3817 raft_consensus.cc:2454] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 1.
I20260621 07:03:55.253661 3956 raft_consensus.cc:2454] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 1.
I20260621 07:03:55.254047 3608 leader_election.cc:302] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb4db43cd9747e5addd015a454dbb6a, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:55.254362 4031 raft_consensus.cc:2781] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:55.254801 4031 raft_consensus.cc:686] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 1 LEADER]: Becoming Leader. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Running, Role: LEADER
I20260621 07:03:55.255260 4031 consensus_queue.cc:237] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } }
I20260621 07:03:55.260283 3521 catalog_manager.cc:5495] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba reported cstate change: term changed from 0 to 1, leader changed from <none> to fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193). New cstate: current_term: 1 leader_uuid: "fa1225def67f448b8dd4bfa3378c3aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:03:55 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025433, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:55.363144 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.349798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.1.188.254:35847)
0621 07:03:55.349979 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.349984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.350520 (+ 536us) server_negotiation.cc:407] Connection header received
0621 07:03:55.350728 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.350733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.350810 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.350913 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:55.352148 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.352903 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.354133 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.354556 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.354702 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.358049 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:55.358054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:55.358057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:55.358110 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:55.360505 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:55.361173 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:55.361178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:55.361188 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:55.361307 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:55.361669 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:55.361673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:55.361676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:55.361953 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:55.362109 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.362519 (+ 410us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.362831 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":62}
I20260621 07:03:55.363183 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.349834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35847 (local address 127.0.0.1:34192)
0621 07:03:55.350414 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.350430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.350602 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.350936 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.350940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.351710 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:03:55.351977 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.351999 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.353037 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.353044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.353988 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.353999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.354745 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.354750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.354909 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.355558 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:03:55.355574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:03:55.357874 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:03:55.360643 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:55.360654 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:55.360662 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:55.361047 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:55.361420 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:03:55.361425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:03:55.361429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:03:55.361553 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:03:55.362086 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:03:55.362094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:03:55.362269 (+ 175us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.362436 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.362718 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
I20260621 07:03:55.368394 3521 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:34192:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:03:55.369380 3521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:55.376320 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 752f3a31cb4c48ec8d921e91526c3882 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc961cc347446fbb38900dfbe302718]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:55.376168 3658 tablet_service.cc:1467] Processing CreateTablet for tablet 752f3a31cb4c48ec8d921e91526c3882 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc961cc347446fbb38900dfbe302718]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:55.377179 3658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752f3a31cb4c48ec8d921e91526c3882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.377219 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752f3a31cb4c48ec8d921e91526c3882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.376783 3936 tablet_service.cc:1467] Processing CreateTablet for tablet 752f3a31cb4c48ec8d921e91526c3882 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cc961cc347446fbb38900dfbe302718]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:03:55.377640 3936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 752f3a31cb4c48ec8d921e91526c3882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.383983 4025 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: Bootstrap starting.
I20260621 07:03:55.384434 4026 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: Bootstrap starting.
I20260621 07:03:55.384434 4029 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: Bootstrap starting.
I20260621 07:03:55.385722 4025 tablet_bootstrap.cc:654] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.386022 4029 tablet_bootstrap.cc:654] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.386022 4026 tablet_bootstrap.cc:654] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.387822 4025 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: No bootstrap required, opened a new log
I20260621 07:03:55.387979 4025 ts_tablet_manager.cc:1397] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 07:03:55.388567 4025 raft_consensus.cc:348] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.388808 4025 raft_consensus.cc:374] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.388900 4025 raft_consensus.cc:729] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.389032 4026 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: No bootstrap required, opened a new log
I20260621 07:03:55.389182 4026 ts_tablet_manager.cc:1397] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 07:03:55.389134 4025 consensus_queue.cc:260] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.389478 4029 tablet_bootstrap.cc:492] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: No bootstrap required, opened a new log
I20260621 07:03:55.389616 4029 ts_tablet_manager.cc:1397] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260621 07:03:55.389711 4025 ts_tablet_manager.cc:1428] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:03:55.390036 4032 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.390041 4026 raft_consensus.cc:348] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.390295 4026 raft_consensus.cc:374] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.390319 4032 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.390404 4026 raft_consensus.cc:729] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.390316 4029 raft_consensus.cc:348] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.390537 4029 raft_consensus.cc:374] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.390643 4029 raft_consensus.cc:729] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc28f76126145baa703162ff44c54db, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.390689 4026 consensus_queue.cc:260] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.390857 4029 consensus_queue.cc:260] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.391160 4031 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.391362 4026 ts_tablet_manager.cc:1428] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 07:03:55.391440 4029 ts_tablet_manager.cc:1428] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 07:03:55.391491 4031 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.391655 4033 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.392035 4033 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.552937 4032 raft_consensus.cc:482] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:55.553242 4032 raft_consensus.cc:504] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.555234 4032 leader_election.cc:288] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), abc28f76126145baa703162ff44c54db (127.1.188.194:39863)
I20260621 07:03:55.562495 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.555801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:58553 (local address 127.1.188.194:39863)
0621 07:03:55.556031 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.556040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.556972 (+ 932us) server_negotiation.cc:407] Connection header received
0621 07:03:55.557126 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.557133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.557225 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.557345 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.558025 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.558901 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.560704 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.561519 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.561697 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.561778 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.562025 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.562154 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260621 07:03:55.562649 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.555645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:53525 (local address 127.1.188.193:43689)
0621 07:03:55.556672 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.556678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.556699 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:03:55.556784 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.556790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.556904 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.557078 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:55.557739 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.558686 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.560642 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.561525 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.561728 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.561820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.562140 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.562307 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":723,"threads_started":1}
I20260621 07:03:55.562904 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.555592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39863 (local address 127.1.188.195:58553)
0621 07:03:55.556829 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.556845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.556982 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.557416 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.557419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.557439 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.557839 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.557852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.559045 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.559051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.560545 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.560556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.561684 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.561690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.561863 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.561888 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.561999 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.562129 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":46,"thread_start_us":925,"threads_started":1}
I20260621 07:03:55.562986 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.555484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:43689 (local address 127.1.188.195:53525)
0621 07:03:55.556027 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.556048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.556216 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.557121 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.557127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.557160 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:55.557502 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.557518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.558853 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.558860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.560457 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.560470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.561772 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.561779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.561973 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.561997 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.562114 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.562268 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":6,"thread_start_us":563,"threads_started":1}
I20260621 07:03:55.563666 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752f3a31cb4c48ec8d921e91526c3882" candidate_uuid: "0bb4db43cd9747e5addd015a454dbb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db" is_pre_election: true
I20260621 07:03:55.564013 3817 raft_consensus.cc:2454] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb4db43cd9747e5addd015a454dbb6a in term 0.
I20260621 07:03:55.564163 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752f3a31cb4c48ec8d921e91526c3882" candidate_uuid: "0bb4db43cd9747e5addd015a454dbb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba" is_pre_election: true
I20260621 07:03:55.564610 3887 leader_election.cc:302] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb4db43cd9747e5addd015a454dbb6a, abc28f76126145baa703162ff44c54db; no voters:
I20260621 07:03:55.564716 3678 raft_consensus.cc:2454] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb4db43cd9747e5addd015a454dbb6a in term 0.
I20260621 07:03:55.565127 4032 raft_consensus.cc:2781] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:55.565284 4032 raft_consensus.cc:482] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:55.565433 4032 raft_consensus.cc:3037] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.568430 4032 raft_consensus.cc:504] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.569134 4032 leader_election.cc:288] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [CANDIDATE]: Term 1 election: Requested vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), abc28f76126145baa703162ff44c54db (127.1.188.194:39863)
I20260621 07:03:55.569628 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752f3a31cb4c48ec8d921e91526c3882" candidate_uuid: "0bb4db43cd9747e5addd015a454dbb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba"
I20260621 07:03:55.569638 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "752f3a31cb4c48ec8d921e91526c3882" candidate_uuid: "0bb4db43cd9747e5addd015a454dbb6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db"
I20260621 07:03:55.569831 3678 raft_consensus.cc:3037] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.569892 3817 raft_consensus.cc:3037] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:55.572240 3678 raft_consensus.cc:2454] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb4db43cd9747e5addd015a454dbb6a in term 1.
I20260621 07:03:55.572240 3817 raft_consensus.cc:2454] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb4db43cd9747e5addd015a454dbb6a in term 1.
I20260621 07:03:55.572672 3889 leader_election.cc:302] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb4db43cd9747e5addd015a454dbb6a, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:55.573071 4032 raft_consensus.cc:2781] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:55.573796 4032 raft_consensus.cc:686] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 LEADER]: Becoming Leader. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Running, Role: LEADER
I20260621 07:03:55.574388 4032 consensus_queue.cc:237] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.578133 4050 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: TxnStatusTablet state changed. Reason: New leader 0bb4db43cd9747e5addd015a454dbb6a. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.578562 4050 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:55.579264 4052 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:03:55.579608 4052 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:03:55.579886 3519 catalog_manager.cc:5495] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195). New cstate: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } health_report { overall_health: HEALTHY } } }
I20260621 07:03:55.631572 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.624107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.0.0.1:59250)
0621 07:03:55.624384 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.624402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.624591 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.625198 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.625203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.625218 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:55.625457 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.625469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.626996 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.627003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.628275 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.628287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.629133 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.629138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.629405 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.629480 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.630837 (+ 1357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.630857 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.631048 (+ 191us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.631251 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260621 07:03:55.631620 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.624054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.1.188.195:38035)
0621 07:03:55.624275 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.624282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.624493 (+ 211us) server_negotiation.cc:407] Connection header received
0621 07:03:55.624822 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.624830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.624966 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.625119 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:55.625677 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.626810 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.628483 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.628963 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.629121 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.629743 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.630624 (+ 881us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.630781 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.631102 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.631229 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:03:55.641691 3678 raft_consensus.cc:1261] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Refusing update from remote peer 0bb4db43cd9747e5addd015a454dbb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:55.642725 4050 consensus_queue.cc:1035] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:55.643091 3817 raft_consensus.cc:1261] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Refusing update from remote peer 0bb4db43cd9747e5addd015a454dbb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:55.643901 4050 consensus_queue.cc:1035] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:55.652050 4031 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: TxnStatusTablet state changed. Reason: New leader 0bb4db43cd9747e5addd015a454dbb6a. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.652426 4031 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.658272 4056 mvcc.cc:205] Tried to move back new op lower bound from 7299176184376356864 to 7299176184117252096. Current Snapshot: MvccSnapshot[applied={T|T < 7299176184376356864}]
I20260621 07:03:55.658841 4032 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.659277 4032 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:55.670691 4033 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: TxnStatusTablet state changed. Reason: New leader 0bb4db43cd9747e5addd015a454dbb6a. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.671125 4033 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.672891 4050 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.673133 4050 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P 0bb4db43cd9747e5addd015a454dbb6a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:03:55.673439 4031 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.673738 4031 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P fa1225def67f448b8dd4bfa3378c3aba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.678351 4033 tablet_replica.cc:439] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bb4db43cd9747e5addd015a454dbb6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } } }
I20260621 07:03:55.678604 4033 tablet_replica.cc:442] T 752f3a31cb4c48ec8d921e91526c3882 P abc28f76126145baa703162ff44c54db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:03:55.686582 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.679666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.1.188.193:43689)
0621 07:03:55.679862 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.679867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.681008 (+ 1141us) server_negotiation.cc:407] Connection header received
0621 07:03:55.681286 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.681294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.681381 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.681597 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:55.682006 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.682804 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.684317 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.684760 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.684895 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.685356 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.685958 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.686115 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.686180 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.686265 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:03:55.704698 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.698272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:51309 (local address 127.1.188.195:38035)
0621 07:03:55.698611 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.698619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.699023 (+ 404us) server_negotiation.cc:407] Connection header received
0621 07:03:55.699173 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.699179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.699288 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.699446 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:55.699888 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.701159 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.702590 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.703309 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.703505 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.703944 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.704105 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.704198 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.704293 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.704375 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 07:03:55.704697 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.698110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.1.188.193:51309)
0621 07:03:55.698898 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.698930 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.699059 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.699459 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.699463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.699473 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:55.699752 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.699766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.701279 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.701283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.702455 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.702467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.703549 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.703553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.703688 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.703745 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.704200 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.704208 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.704267 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.704374 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":475,"threads_started":1}
I20260621 07:03:55.715332 3816 raft_consensus.cc:1261] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Refusing update from remote peer fa1225def67f448b8dd4bfa3378c3aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:55.715796 3956 raft_consensus.cc:1261] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Refusing update from remote peer fa1225def67f448b8dd4bfa3378c3aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:55.716037 4036 consensus_queue.cc:1035] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:55.716485 4031 consensus_queue.cc:1035] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:55.725289 4055 mvcc.cc:205] Tried to move back new op lower bound from 7299176184683716608 to 7299176182809636864. Current Snapshot: MvccSnapshot[applied={T|T < 7299176184683716608}]
I20260621 07:03:55.726027 4058 mvcc.cc:205] Tried to move back new op lower bound from 7299176184683716608 to 7299176182809636864. Current Snapshot: MvccSnapshot[applied={T|T < 7299176184683716608}]
I20260621 07:03:55.755077 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.748136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:45865 (local address 127.1.188.193:43689)
0621 07:03:55.748365 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.748376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:55.749088 (+ 712us) server_negotiation.cc:407] Connection header received
0621 07:03:55.749293 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:55.749299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:55.749407 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:55.749506 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:55.750207 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.751444 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.753090 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.753671 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.753803 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.754257 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.754391 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.754488 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:55.754637 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 07:03:55.754724 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:03:55.755615 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.748083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:43689 (local address 127.1.188.195:45865)
0621 07:03:55.748950 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:55.748973 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:55.749148 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:55.749533 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:55.749540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:55.749559 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:55.750031 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.750043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.751585 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.751590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.752939 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:55.752952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:55.753871 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:55.753878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:55.754015 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:55.754082 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:55.754485 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:55.754498 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 07:03:55.754604 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:03:55.754754 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":529,"threads_started":1}
I20260621 07:03:55.796121 3520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:55.796518 3520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:55.801312 3520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Sending DeleteTablet for 3 replicas of tablet 29422781001c4e2d9b7560ed730b1f04
I20260621 07:03:55.802695 3658 tablet_service.cc:1514] Processing DeleteTablet for tablet 29422781001c4e2d9b7560ed730b1f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:55 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260621 07:03:55.803290 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 29422781001c4e2d9b7560ed730b1f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:55 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:42176
I20260621 07:03:55.805915 3519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:55.806697 3519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:55.808724 3936 tablet_service.cc:1514] Processing DeleteTablet for tablet 29422781001c4e2d9b7560ed730b1f04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:55 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59248
I20260621 07:03:55.816284 3657 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 (DEFAULT_TABLE table=test-table [id=699006763e4a44209e251ef4977927d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.817108 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6d95826ca430d8afdeb7d11d7acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.817165 3935 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 (DEFAULT_TABLE table=test-table [id=699006763e4a44209e251ef4977927d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.817782 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6d95826ca430d8afdeb7d11d7acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.820417 3796 tablet_service.cc:1467] Processing CreateTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 (DEFAULT_TABLE table=test-table [id=699006763e4a44209e251ef4977927d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:55.821031 3796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db6d95826ca430d8afdeb7d11d7acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:55.826637 4076 tablet_replica.cc:330] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: stopping tablet replica
I20260621 07:03:55.826943 4025 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Bootstrap starting.
I20260621 07:03:55.828323 4076 raft_consensus.cc:2229] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:03:55.828799 4025 tablet_bootstrap.cc:654] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.829188 4076 raft_consensus.cc:2258] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:55.832824 4029 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Bootstrap starting.
I20260621 07:03:55.834470 4076 ts_tablet_manager.cc:1905] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:55.834735 4029 tablet_bootstrap.cc:654] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.835011 4078 tablet_replica.cc:330] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: stopping tablet replica
I20260621 07:03:55.835587 4078 raft_consensus.cc:2229] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:55.835978 4078 raft_consensus.cc:2258] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:55.836601 4026 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Bootstrap starting.
I20260621 07:03:55.839056 4026 tablet_bootstrap.cc:654] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:55.846132 4025 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: No bootstrap required, opened a new log
I20260621 07:03:55.846318 4025 ts_tablet_manager.cc:1397] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
I20260621 07:03:55.847252 4025 raft_consensus.cc:348] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.847605 4025 raft_consensus.cc:374] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.847714 4025 raft_consensus.cc:729] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.848096 4025 consensus_queue.cc:260] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.848752 4025 ts_tablet_manager.cc:1428] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:03:55.849408 4077 tablet_replica.cc:330] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: stopping tablet replica
I20260621 07:03:55.849953 4078 ts_tablet_manager.cc:1905] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:55.849917 4077 raft_consensus.cc:2229] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:55.850368 4077 raft_consensus.cc:2258] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:55.852699 4076 ts_tablet_manager.cc:1918] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:55.852897 4076 log.cc:1196] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/29422781001c4e2d9b7560ed730b1f04
I20260621 07:03:55.853521 4076 ts_tablet_manager.cc:1939] T 29422781001c4e2d9b7560ed730b1f04 P fa1225def67f448b8dd4bfa3378c3aba: Deleting consensus metadata
I20260621 07:03:55.855494 4026 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: No bootstrap required, opened a new log
I20260621 07:03:55.855669 4026 ts_tablet_manager.cc:1397] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20260621 07:03:55.855625 3507 catalog_manager.cc:4841] TS fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689): tablet 29422781001c4e2d9b7560ed730b1f04 (table test-table [id=654a234f951e4cbdb3596717dc850109]) successfully deleted
I20260621 07:03:55.856410 4029 tablet_bootstrap.cc:492] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: No bootstrap required, opened a new log
I20260621 07:03:55.856659 4029 ts_tablet_manager.cc:1397] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.009s
I20260621 07:03:55.856581 4026 raft_consensus.cc:348] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.857019 4026 raft_consensus.cc:374] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.857162 4026 raft_consensus.cc:729] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.857512 4026 consensus_queue.cc:260] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.858347 4026 ts_tablet_manager.cc:1428] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 07:03:55.858685 4078 ts_tablet_manager.cc:1918] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:55.858848 4078 log.cc:1196] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/29422781001c4e2d9b7560ed730b1f04
I20260621 07:03:55.859472 4078 ts_tablet_manager.cc:1939] T 29422781001c4e2d9b7560ed730b1f04 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting consensus metadata
I20260621 07:03:55.861434 3505 catalog_manager.cc:4841] TS 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035): tablet 29422781001c4e2d9b7560ed730b1f04 (table test-table [id=654a234f951e4cbdb3596717dc850109]) successfully deleted
I20260621 07:03:55.862591 4029 raft_consensus.cc:348] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.862920 4029 raft_consensus.cc:374] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:55.863022 4029 raft_consensus.cc:729] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc28f76126145baa703162ff44c54db, State: Initialized, Role: FOLLOWER
I20260621 07:03:55.863291 4029 consensus_queue.cc:260] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.864077 4029 ts_tablet_manager.cc:1428] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260621 07:03:55.865442 4077 ts_tablet_manager.cc:1905] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:55.873483 4077 ts_tablet_manager.cc:1918] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:55.873678 4077 log.cc:1196] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/29422781001c4e2d9b7560ed730b1f04
I20260621 07:03:55.874251 4077 ts_tablet_manager.cc:1939] T 29422781001c4e2d9b7560ed730b1f04 P abc28f76126145baa703162ff44c54db: Deleting consensus metadata
I20260621 07:03:55.876420 3505 catalog_manager.cc:4841] TS abc28f76126145baa703162ff44c54db (127.1.188.194:39863): tablet 29422781001c4e2d9b7560ed730b1f04 (table test-table [id=654a234f951e4cbdb3596717dc850109]) successfully deleted
I20260621 07:03:55.992982 4033 raft_consensus.cc:482] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:55.993347 4033 raft_consensus.cc:504] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:55.995790 4033 leader_election.cc:288] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:56.016649 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.996477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:35857 (local address 127.1.188.193:43689)
0621 07:03:55.996731 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.996738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:56.007361 (+ 10623us) server_negotiation.cc:407] Connection header received
0621 07:03:56.007998 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:56.008005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:56.008099 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:56.008168 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:56.009104 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.010537 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.013751 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.015151 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.015328 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.015446 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:56.016079 (+ 633us) server_negotiation.cc:299] Negotiation successful
0621 07:03:56.016249 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 07:03:56.018091 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.996026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:43689 (local address 127.1.188.194:35857)
0621 07:03:56.007194 (+ 11168us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:56.007226 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:56.007785 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:56.008403 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:56.008409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:56.008441 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:56.008900 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.008915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.010737 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.010743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.013557 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.013572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.015666 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.015673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.015867 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.015906 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:03:56.017391 (+ 1485us) client_negotiation.cc:240] Negotiation successful
0621 07:03:56.017579 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10978,"mutex_wait_us":45,"thread_start_us":11362,"threads_started":1}
I20260621 07:03:56.019055 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6d95826ca430d8afdeb7d11d7acb7" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba" is_pre_election: true
I20260621 07:03:56.019387 3678 raft_consensus.cc:2454] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 0.
I20260621 07:03:56.020813 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.996790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:53291 (local address 127.1.188.195:38035)
0621 07:03:55.996998 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:55.997005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:56.009798 (+ 12793us) server_negotiation.cc:407] Connection header received
0621 07:03:56.010363 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:56.010371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:56.010480 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:56.010564 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:03:56.011812 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.013042 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.017732 (+ 4690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.019284 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.019475 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.019567 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:56.020305 (+ 738us) server_negotiation.cc:299] Negotiation successful
0621 07:03:56.020458 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 07:03:56.020844 3750 leader_election.cc:302] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc28f76126145baa703162ff44c54db, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:56.021359 4033 raft_consensus.cc:2781] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:56.021518 4033 raft_consensus.cc:482] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:56.021627 4033 raft_consensus.cc:3037] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:56.024293 4083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:55.996327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.1.188.194:53291)
0621 07:03:56.009560 (+ 13233us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:56.009583 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:56.010152 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:56.010821 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:56.010827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:56.010858 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:03:56.011564 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.011581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.013257 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.013264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.017537 (+ 4273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.017553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.019812 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.019819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.020037 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.020078 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 07:03:56.021103 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0621 07:03:56.021326 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13085,"mutex_wait_us":83,"thread_start_us":13519,"threads_started":1}
I20260621 07:03:56.025295 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6d95826ca430d8afdeb7d11d7acb7" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a" is_pre_election: true
I20260621 07:03:56.025347 4033 raft_consensus.cc:504] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:56.025626 3956 raft_consensus.cc:2454] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 0.
I20260621 07:03:56.026700 4033 leader_election.cc:288] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 election: Requested vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:56.026685 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6d95826ca430d8afdeb7d11d7acb7" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba"
I20260621 07:03:56.026978 3678 raft_consensus.cc:3037] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:56.027477 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db6d95826ca430d8afdeb7d11d7acb7" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a"
I20260621 07:03:56.027719 3956 raft_consensus.cc:3037] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:56.030678 3678 raft_consensus.cc:2454] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 1.
I20260621 07:03:56.031119 3956 raft_consensus.cc:2454] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 1.
I20260621 07:03:56.031209 3750 leader_election.cc:302] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc28f76126145baa703162ff44c54db, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:56.032223 4084 raft_consensus.cc:2781] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:56.032447 4084 raft_consensus.cc:686] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 1 LEADER]: Becoming Leader. State: Replica: abc28f76126145baa703162ff44c54db, State: Running, Role: LEADER
I20260621 07:03:56.033046 4084 consensus_queue.cc:237] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:56.038277 3520 catalog_manager.cc:5495] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db reported cstate change: term changed from 0 to 1, leader changed from <none> to abc28f76126145baa703162ff44c54db (127.1.188.194). New cstate: current_term: 1 leader_uuid: "abc28f76126145baa703162ff44c54db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } health_report { overall_health: UNKNOWN } } }
I20260621 07:03:56.090322 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:56.081515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.1.188.194:39863)
0621 07:03:56.082720 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:56.082728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:56.082749 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:03:56.082874 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:56.082880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:56.083018 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:56.083228 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:56.083718 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.084996 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.086964 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.087764 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.087926 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.088460 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:56.089428 (+ 968us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:56.089615 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:56.089732 (+ 117us) server_negotiation.cc:299] Negotiation successful
0621 07:03:56.089849 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":957,"threads_started":1}
I20260621 07:03:56.116434 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:56.105973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55253 (local address 127.1.188.195:38035)
0621 07:03:56.106195 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:56.106203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:56.106455 (+ 252us) server_negotiation.cc:407] Connection header received
0621 07:03:56.106840 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:56.106848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:56.106988 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:56.107093 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:56.108020 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.108802 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.113049 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.114105 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.114246 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.115319 (+ 1073us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:56.115565 (+ 246us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:56.115644 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:56.115910 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 07:03:56.116026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:03:56.118081 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:56.105440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38035 (local address 127.1.188.194:55253)
0621 07:03:56.106298 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:56.106324 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:56.106622 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:56.107230 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:56.107239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:56.107255 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:56.107774 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.107790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.108974 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.108981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.112823 (+ 3842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.112849 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.114519 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.114526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.114742 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.114846 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:56.115743 (+ 897us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:56.115756 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 07:03:56.117103 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0621 07:03:56.117294 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":765,"threads_started":1}
I20260621 07:03:56.140334 3678 raft_consensus.cc:1261] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Refusing update from remote peer abc28f76126145baa703162ff44c54db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:56.141319 4033 consensus_queue.cc:1035] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:56.145828 3956 raft_consensus.cc:1261] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Refusing update from remote peer abc28f76126145baa703162ff44c54db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:56.146697 4084 consensus_queue.cc:1035] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260621 07:03:56.202767 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:56.193724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39863 (local address 127.1.188.195:40343)
0621 07:03:56.194189 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0621 07:03:56.194213 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:03:56.194381 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:03:56.195516 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:03:56.195524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:03:56.195543 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:03:56.195796 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.195812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.197614 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.197620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.199308 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:03:56.199336 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.201166 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.201173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:03:56.201417 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.201486 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:03:56.202030 (+ 544us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:03:56.202046 (+ 16us) client_negotiation.cc:769] Sending connection context
0621 07:03:56.202142 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 07:03:56.202311 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260621 07:03:56.203624 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:56.194823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:40343 (local address 127.1.188.194:39863)
0621 07:03:56.195059 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:56.195066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:56.195089 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:03:56.195181 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:56.195188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:56.195297 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:56.195974 (+ 677us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:56.196166 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.197405 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.199932 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:56.200762 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:56.201033 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:56.201712 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:56.201873 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:56.201973 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:56.203132 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0621 07:03:56.203228 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260621 07:03:56.204948 4058 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:03:57.815723 3519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:57.816097 3519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:57.820711 3519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Sending DeleteTablet for 3 replicas of tablet 1db6d95826ca430d8afdeb7d11d7acb7
I20260621 07:03:57.821570 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:57 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:42176
I20260621 07:03:57.821678 3935 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:57 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59248
I20260621 07:03:57.821766 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db6d95826ca430d8afdeb7d11d7acb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:57 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260621 07:03:57.832744 4116 tablet_replica.cc:330] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: stopping tablet replica
I20260621 07:03:57.833125 4116 raft_consensus.cc:2229] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:03:57.834168 4116 raft_consensus.cc:2258] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:57.834916 4118 tablet_replica.cc:330] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: stopping tablet replica
I20260621 07:03:57.835299 4117 tablet_replica.cc:330] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: stopping tablet replica
I20260621 07:03:57.835544 4117 raft_consensus.cc:2229] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:57.835888 4117 raft_consensus.cc:2258] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:57.836174 4116 ts_tablet_manager.cc:1905] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:57.836228 4118 raft_consensus.cc:2229] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:57.836504 4118 raft_consensus.cc:2258] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:57.837342 4117 ts_tablet_manager.cc:1905] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:57.837580 4118 ts_tablet_manager.cc:1905] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:57.843957 4116 ts_tablet_manager.cc:1918] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:03:57.844117 4116 log.cc:1196] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/1db6d95826ca430d8afdeb7d11d7acb7
I20260621 07:03:57.844646 4116 ts_tablet_manager.cc:1939] T 1db6d95826ca430d8afdeb7d11d7acb7 P abc28f76126145baa703162ff44c54db: Deleting consensus metadata
I20260621 07:03:57.844678 4118 ts_tablet_manager.cc:1918] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:03:57.844812 4118 log.cc:1196] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/1db6d95826ca430d8afdeb7d11d7acb7
I20260621 07:03:57.845242 4118 ts_tablet_manager.cc:1939] T 1db6d95826ca430d8afdeb7d11d7acb7 P fa1225def67f448b8dd4bfa3378c3aba: Deleting consensus metadata
I20260621 07:03:57.846110 4117 ts_tablet_manager.cc:1918] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:03:57.846189 3505 catalog_manager.cc:4841] TS abc28f76126145baa703162ff44c54db (127.1.188.194:39863): tablet 1db6d95826ca430d8afdeb7d11d7acb7 (table test-table [id=699006763e4a44209e251ef4977927d8]) successfully deleted
I20260621 07:03:57.846307 4117 log.cc:1196] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/1db6d95826ca430d8afdeb7d11d7acb7
I20260621 07:03:57.846465 3507 catalog_manager.cc:4841] TS fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689): tablet 1db6d95826ca430d8afdeb7d11d7acb7 (table test-table [id=699006763e4a44209e251ef4977927d8]) successfully deleted
I20260621 07:03:57.846937 4117 ts_tablet_manager.cc:1939] T 1db6d95826ca430d8afdeb7d11d7acb7 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting consensus metadata
I20260621 07:03:57.848230 3505 catalog_manager.cc:4841] TS 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035): tablet 1db6d95826ca430d8afdeb7d11d7acb7 (table test-table [id=699006763e4a44209e251ef4977927d8]) successfully deleted
Jun 21 07:03:57 dist-test-slave-hlq1 krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:03:57 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:03:57.888958 4126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:57.870336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.1.188.254:35847)
0621 07:03:57.870983 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:57.870988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:57.871013 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:03:57.871215 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:57.871219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:57.871295 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:57.871480 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:03:57.872884 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:57.873917 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:57.875243 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:57.875777 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:57.875900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:57.883298 (+ 7398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:03:57.883305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:03:57.883309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:03:57.883357 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:03:57.886324 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:57.886974 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:57.886979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:57.886982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:57.887149 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:03:57.887619 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:03:57.887623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:03:57.887625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:03:57.887951 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:03:57.888153 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:57.888429 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 07:03:57.888622 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":325,"threads_started":1}
I20260621 07:03:57.893093 3519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:57.893774 3519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:57.900939 3657 tablet_service.cc:1467] Processing CreateTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 (DEFAULT_TABLE table=test-table [id=5d4625834c3b489fb326513f43e1dacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:57.901028 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 (DEFAULT_TABLE table=test-table [id=5d4625834c3b489fb326513f43e1dacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:57.901188 3935 tablet_service.cc:1467] Processing CreateTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 (DEFAULT_TABLE table=test-table [id=5d4625834c3b489fb326513f43e1dacc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:57.901517 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69373cf8860c444fbe6b3eb8471cda96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:57.901541 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69373cf8860c444fbe6b3eb8471cda96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:57.901685 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69373cf8860c444fbe6b3eb8471cda96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:57.907572 4127 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Bootstrap starting.
I20260621 07:03:57.908838 4129 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Bootstrap starting.
I20260621 07:03:57.909695 4127 tablet_bootstrap.cc:654] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:57.910648 4129 tablet_bootstrap.cc:654] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:57.910804 4128 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Bootstrap starting.
I20260621 07:03:57.912470 4127 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: No bootstrap required, opened a new log
I20260621 07:03:57.912645 4127 ts_tablet_manager.cc:1397] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 07:03:57.912906 4128 tablet_bootstrap.cc:654] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:57.913208 4129 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: No bootstrap required, opened a new log
I20260621 07:03:57.913362 4129 ts_tablet_manager.cc:1397] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:03:57.913622 4127 raft_consensus.cc:348] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.913934 4127 raft_consensus.cc:374] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:57.914037 4127 raft_consensus.cc:729] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Initialized, Role: FOLLOWER
I20260621 07:03:57.914311 4127 consensus_queue.cc:260] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.914278 4129 raft_consensus.cc:348] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.914604 4129 raft_consensus.cc:374] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:57.914727 4129 raft_consensus.cc:729] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc28f76126145baa703162ff44c54db, State: Initialized, Role: FOLLOWER
I20260621 07:03:57.915035 4129 consensus_queue.cc:260] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.915720 4127 ts_tablet_manager.cc:1428] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260621 07:03:57.915728 4129 ts_tablet_manager.cc:1428] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 07:03:57.917511 4128 tablet_bootstrap.cc:492] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: No bootstrap required, opened a new log
I20260621 07:03:57.917688 4128 ts_tablet_manager.cc:1397] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:03:57.918613 4128 raft_consensus.cc:348] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.918910 4128 raft_consensus.cc:374] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:57.919020 4128 raft_consensus.cc:729] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Initialized, Role: FOLLOWER
I20260621 07:03:57.919277 4128 consensus_queue.cc:260] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:57.919930 4128 ts_tablet_manager.cc:1428] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 07:03:58.143544 4133 raft_consensus.cc:482] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:58.143843 4133 raft_consensus.cc:504] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.144677 4133 leader_election.cc:288] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abc28f76126145baa703162ff44c54db (127.1.188.194:39863), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:58.145406 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69373cf8860c444fbe6b3eb8471cda96" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db" is_pre_election: true
I20260621 07:03:58.145464 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69373cf8860c444fbe6b3eb8471cda96" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a" is_pre_election: true
I20260621 07:03:58.145748 3817 raft_consensus.cc:2454] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 0.
I20260621 07:03:58.145778 3956 raft_consensus.cc:2454] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 0.
I20260621 07:03:58.146307 3608 leader_election.cc:302] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc28f76126145baa703162ff44c54db, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:58.146607 4133 raft_consensus.cc:2781] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:58.146747 4133 raft_consensus.cc:482] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:58.146863 4133 raft_consensus.cc:3037] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.149233 4133 raft_consensus.cc:504] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.149748 4133 leader_election.cc:288] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 election: Requested vote from peers abc28f76126145baa703162ff44c54db (127.1.188.194:39863), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:58.150276 3817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69373cf8860c444fbe6b3eb8471cda96" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc28f76126145baa703162ff44c54db"
I20260621 07:03:58.150318 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69373cf8860c444fbe6b3eb8471cda96" candidate_uuid: "fa1225def67f448b8dd4bfa3378c3aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a"
I20260621 07:03:58.150501 3817 raft_consensus.cc:3037] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.150540 3956 raft_consensus.cc:3037] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.152783 3817 raft_consensus.cc:2454] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 1.
I20260621 07:03:58.152803 3956 raft_consensus.cc:2454] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1225def67f448b8dd4bfa3378c3aba in term 1.
I20260621 07:03:58.153195 3608 leader_election.cc:302] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc28f76126145baa703162ff44c54db, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:58.153523 4133 raft_consensus.cc:2781] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:58.154016 4133 raft_consensus.cc:686] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 1 LEADER]: Becoming Leader. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Running, Role: LEADER
I20260621 07:03:58.154309 4133 consensus_queue.cc:237] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.157371 3520 catalog_manager.cc:5495] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba reported cstate change: term changed from 0 to 1, leader changed from <none> to fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193). New cstate: current_term: 1 leader_uuid: "fa1225def67f448b8dd4bfa3378c3aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } health_report { overall_health: UNKNOWN } } }
I20260621 07:03:58.220093 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:58.214256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.1.188.193:43689)
0621 07:03:58.214844 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:58.214848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:58.214867 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:03:58.214921 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:58.214924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:58.214990 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:58.215115 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:58.215568 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:58.216578 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:58.218076 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:58.218551 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:58.218690 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:58.219163 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:58.219294 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:58.219394 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:58.219592 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 07:03:58.219679 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
I20260621 07:03:58.229981 3956 raft_consensus.cc:1261] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Refusing update from remote peer fa1225def67f448b8dd4bfa3378c3aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:58.230304 3817 raft_consensus.cc:1261] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Refusing update from remote peer fa1225def67f448b8dd4bfa3378c3aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:58.230649 4133 consensus_queue.cc:1035] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:58.230928 4134 consensus_queue.cc:1035] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:58.276571 3520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:58.276820 3520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:03:58.279901 3520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Sending DeleteTablet for 3 replicas of tablet 69373cf8860c444fbe6b3eb8471cda96
I20260621 07:03:58.280750 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:58 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260621 07:03:58.280750 3935 tablet_service.cc:1514] Processing DeleteTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:58 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59248
I20260621 07:03:58.280782 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 69373cf8860c444fbe6b3eb8471cda96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:03:58 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:42176
I20260621 07:03:58.281087 4118 tablet_replica.cc:330] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: stopping tablet replica
I20260621 07:03:58.281086 4117 tablet_replica.cc:330] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: stopping tablet replica
I20260621 07:03:58.281092 4116 tablet_replica.cc:330] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: stopping tablet replica
I20260621 07:03:58.281301 4118 raft_consensus.cc:2229] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:03:58.281301 4117 raft_consensus.cc:2229] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:58.281307 4116 raft_consensus.cc:2229] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:03:58.281502 4117 raft_consensus.cc:2258] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:58.281520 4116 raft_consensus.cc:2258] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:58.281649 4118 raft_consensus.cc:2258] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:03:58.281929 3520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:03:58.282706 3520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:03:58.282891 4117 ts_tablet_manager.cc:1905] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:58.283016 4116 ts_tablet_manager.cc:1905] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:58.283491 4118 ts_tablet_manager.cc:1905] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:03:58.289348 4116 ts_tablet_manager.cc:1918] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:58.289535 4116 log.cc:1196] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/69373cf8860c444fbe6b3eb8471cda96
I20260621 07:03:58.289866 4117 ts_tablet_manager.cc:1918] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:58.289978 4116 ts_tablet_manager.cc:1939] T 69373cf8860c444fbe6b3eb8471cda96 P abc28f76126145baa703162ff44c54db: Deleting consensus metadata
I20260621 07:03:58.290037 4117 log.cc:1196] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/69373cf8860c444fbe6b3eb8471cda96
I20260621 07:03:58.290493 4117 ts_tablet_manager.cc:1939] T 69373cf8860c444fbe6b3eb8471cda96 P 0bb4db43cd9747e5addd015a454dbb6a: Deleting consensus metadata
I20260621 07:03:58.291420 4118 ts_tablet_manager.cc:1918] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:03:58.291677 4118 log.cc:1196] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/69373cf8860c444fbe6b3eb8471cda96
I20260621 07:03:58.291837 3505 catalog_manager.cc:4841] TS 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035): tablet 69373cf8860c444fbe6b3eb8471cda96 (table test-table [id=5d4625834c3b489fb326513f43e1dacc]) successfully deleted
I20260621 07:03:58.292235 4118 ts_tablet_manager.cc:1939] T 69373cf8860c444fbe6b3eb8471cda96 P fa1225def67f448b8dd4bfa3378c3aba: Deleting consensus metadata
I20260621 07:03:58.293166 3505 catalog_manager.cc:4841] TS abc28f76126145baa703162ff44c54db (127.1.188.194:39863): tablet 69373cf8860c444fbe6b3eb8471cda96 (table test-table [id=5d4625834c3b489fb326513f43e1dacc]) successfully deleted
I20260621 07:03:58.293378 3507 catalog_manager.cc:4841] TS fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689): tablet 69373cf8860c444fbe6b3eb8471cda96 (table test-table [id=5d4625834c3b489fb326513f43e1dacc]) successfully deleted
I20260621 07:03:58.294620 3657 tablet_service.cc:1467] Processing CreateTablet for tablet 2193960a635c414d8f46d7e540167a6f (DEFAULT_TABLE table=test-table [id=6f81770057d74f539a49fcfa9a711cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:58.294819 3797 tablet_service.cc:1467] Processing CreateTablet for tablet 2193960a635c414d8f46d7e540167a6f (DEFAULT_TABLE table=test-table [id=6f81770057d74f539a49fcfa9a711cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:58.295051 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2193960a635c414d8f46d7e540167a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:58.295199 3935 tablet_service.cc:1467] Processing CreateTablet for tablet 2193960a635c414d8f46d7e540167a6f (DEFAULT_TABLE table=test-table [id=6f81770057d74f539a49fcfa9a711cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:03:58.295316 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2193960a635c414d8f46d7e540167a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:58.295648 3935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2193960a635c414d8f46d7e540167a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:03:58.302239 4128 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Bootstrap starting.
I20260621 07:03:58.302315 4127 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Bootstrap starting.
I20260621 07:03:58.302573 4129 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Bootstrap starting.
I20260621 07:03:58.304073 4127 tablet_bootstrap.cc:654] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:58.304142 4128 tablet_bootstrap.cc:654] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:58.304340 4129 tablet_bootstrap.cc:654] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Neither blocks nor log segments found. Creating new log.
I20260621 07:03:58.313761 4128 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: No bootstrap required, opened a new log
I20260621 07:03:58.314010 4128 ts_tablet_manager.cc:1397] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260621 07:03:58.315137 4128 raft_consensus.cc:348] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.315523 4128 raft_consensus.cc:374] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:58.315631 4128 raft_consensus.cc:729] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb4db43cd9747e5addd015a454dbb6a, State: Initialized, Role: FOLLOWER
I20260621 07:03:58.315922 4128 consensus_queue.cc:260] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.316723 4129 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: No bootstrap required, opened a new log
I20260621 07:03:58.316953 4129 ts_tablet_manager.cc:1397] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260621 07:03:58.317263 4127 tablet_bootstrap.cc:492] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: No bootstrap required, opened a new log
I20260621 07:03:58.317479 4127 ts_tablet_manager.cc:1397] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260621 07:03:58.317909 4129 raft_consensus.cc:348] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.318244 4129 raft_consensus.cc:374] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:58.318359 4129 raft_consensus.cc:729] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc28f76126145baa703162ff44c54db, State: Initialized, Role: FOLLOWER
I20260621 07:03:58.318544 4127 raft_consensus.cc:348] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.318639 4129 consensus_queue.cc:260] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.318859 4127 raft_consensus.cc:374] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:03:58.318960 4127 raft_consensus.cc:729] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1225def67f448b8dd4bfa3378c3aba, State: Initialized, Role: FOLLOWER
I20260621 07:03:58.319278 4127 consensus_queue.cc:260] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.319932 4127 ts_tablet_manager.cc:1428] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 07:03:58.320308 4129 ts_tablet_manager.cc:1428] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260621 07:03:58.321024 4128 ts_tablet_manager.cc:1428] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260621 07:03:58.417342 4094 raft_consensus.cc:482] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:03:58.417711 4094 raft_consensus.cc:504] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.418545 4094 leader_election.cc:288] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:58.419349 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2193960a635c414d8f46d7e540167a6f" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a" is_pre_election: true
I20260621 07:03:58.419349 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2193960a635c414d8f46d7e540167a6f" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba" is_pre_election: true
I20260621 07:03:58.419682 3956 raft_consensus.cc:2454] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 0.
I20260621 07:03:58.419682 3678 raft_consensus.cc:2454] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 0.
I20260621 07:03:58.420248 3748 leader_election.cc:302] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb4db43cd9747e5addd015a454dbb6a, abc28f76126145baa703162ff44c54db; no voters:
I20260621 07:03:58.420593 4094 raft_consensus.cc:2781] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:03:58.420718 4094 raft_consensus.cc:482] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:03:58.420836 4094 raft_consensus.cc:3037] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.423476 4094 raft_consensus.cc:504] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.424196 4094 leader_election.cc:288] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 election: Requested vote from peers fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689), 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035)
I20260621 07:03:58.424729 3678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2193960a635c414d8f46d7e540167a6f" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1225def67f448b8dd4bfa3378c3aba"
I20260621 07:03:58.424759 3956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2193960a635c414d8f46d7e540167a6f" candidate_uuid: "abc28f76126145baa703162ff44c54db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb4db43cd9747e5addd015a454dbb6a"
I20260621 07:03:58.424988 3678 raft_consensus.cc:3037] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.425014 3956 raft_consensus.cc:3037] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:03:58.427740 3678 raft_consensus.cc:2454] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 1.
I20260621 07:03:58.428206 3956 raft_consensus.cc:2454] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abc28f76126145baa703162ff44c54db in term 1.
I20260621 07:03:58.428301 3750 leader_election.cc:302] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc28f76126145baa703162ff44c54db, fa1225def67f448b8dd4bfa3378c3aba; no voters:
I20260621 07:03:58.428620 4094 raft_consensus.cc:2781] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:03:58.429149 4094 raft_consensus.cc:686] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 1 LEADER]: Becoming Leader. State: Replica: abc28f76126145baa703162ff44c54db, State: Running, Role: LEADER
I20260621 07:03:58.429517 4094 consensus_queue.cc:237] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } }
I20260621 07:03:58.432348 3521 catalog_manager.cc:5495] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db reported cstate change: term changed from 0 to 1, leader changed from <none> to abc28f76126145baa703162ff44c54db (127.1.188.194). New cstate: current_term: 1 leader_uuid: "abc28f76126145baa703162ff44c54db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abc28f76126145baa703162ff44c54db" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 } health_report { overall_health: UNKNOWN } } }
I20260621 07:03:58.489786 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:03:58.475158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.1.188.194:39863)
0621 07:03:58.484061 (+ 8903us) server_negotiation.cc:206] Beginning negotiation
0621 07:03:58.484069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:03:58.484104 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:03:58.484200 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:03:58.484205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:03:58.484286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:03:58.484432 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:03:58.484926 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:58.486048 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:58.487676 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:03:58.488136 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:03:58.488285 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:03:58.488809 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:03:58.488960 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:03:58.489068 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:03:58.489227 (+ 159us) server_negotiation.cc:299] Negotiation successful
0621 07:03:58.489400 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8712,"thread_start_us":8521,"threads_started":1}
I20260621 07:03:58.501063 3678 raft_consensus.cc:1261] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Refusing update from remote peer abc28f76126145baa703162ff44c54db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:58.501241 3956 raft_consensus.cc:1261] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Refusing update from remote peer abc28f76126145baa703162ff44c54db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:03:58.501773 4145 consensus_queue.cc:1035] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1225def67f448b8dd4bfa3378c3aba" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:03:58.502099 4094 consensus_queue.cc:1035] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb4db43cd9747e5addd015a454dbb6a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:03:58.525673 4098 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 07:04:00.110388 3521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:00.110662 3521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:00.115324 3521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bb5eff83f04f77ac30c336ac583d3b: Sending DeleteTablet for 3 replicas of tablet 2193960a635c414d8f46d7e540167a6f
I20260621 07:04:00.116384 3797 tablet_service.cc:1514] Processing DeleteTablet for tablet 2193960a635c414d8f46d7e540167a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:00 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:42176
I20260621 07:04:00.116420 3657 tablet_service.cc:1514] Processing DeleteTablet for tablet 2193960a635c414d8f46d7e540167a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:00 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:43020
I20260621 07:04:00.116494 3935 tablet_service.cc:1514] Processing DeleteTablet for tablet 2193960a635c414d8f46d7e540167a6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:00 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59248
I20260621 07:04:00.119565 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3592
I20260621 07:04:00.130070 4174 tablet_replica.cc:330] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: stopping tablet replica
I20260621 07:04:00.130760 4176 tablet_replica.cc:330] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: stopping tablet replica
I20260621 07:04:00.131201 4175 tablet_replica.cc:330] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: stopping tablet replica
I20260621 07:04:00.131201 4176 raft_consensus.cc:2229] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:00.131547 4174 raft_consensus.cc:2229] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:00.131541 4175 raft_consensus.cc:2229] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:00.131855 4175 raft_consensus.cc:2258] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:00.132014 4174 raft_consensus.cc:2258] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:00.132263 4176 raft_consensus.cc:2258] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:00.133245 4175 ts_tablet_manager.cc:1905] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:00.133462 4176 ts_tablet_manager.cc:1905] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:00.134052 4174 ts_tablet_manager.cc:1905] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:00.140837 4175 ts_tablet_manager.cc:1918] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:04:00.141024 4175 log.cc:1196] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/2193960a635c414d8f46d7e540167a6f
I20260621 07:04:00.141189 4176 ts_tablet_manager.cc:1918] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:04:00.141426 4176 log.cc:1196] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/2193960a635c414d8f46d7e540167a6f
I20260621 07:04:00.141556 4175 ts_tablet_manager.cc:1939] T 2193960a635c414d8f46d7e540167a6f P fa1225def67f448b8dd4bfa3378c3aba: Deleting consensus metadata
I20260621 07:04:00.141834 4174 ts_tablet_manager.cc:1918] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 07:04:00.141964 4176 ts_tablet_manager.cc:1939] T 2193960a635c414d8f46d7e540167a6f P 0bb4db43cd9747e5addd015a454dbb6a: Deleting consensus metadata
I20260621 07:04:00.142024 4174 log.cc:1196] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/2193960a635c414d8f46d7e540167a6f
I20260621 07:04:00.142589 4174 ts_tablet_manager.cc:1939] T 2193960a635c414d8f46d7e540167a6f P abc28f76126145baa703162ff44c54db: Deleting consensus metadata
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:00.142951 3507 catalog_manager.cc:4841] TS fa1225def67f448b8dd4bfa3378c3aba (127.1.188.193:43689): tablet 2193960a635c414d8f46d7e540167a6f (table test-table [id=6f81770057d74f539a49fcfa9a711cfb]) successfully deleted
I20260621 07:04:00.143277 3505 catalog_manager.cc:4841] TS 0bb4db43cd9747e5addd015a454dbb6a (127.1.188.195:38035): tablet 2193960a635c414d8f46d7e540167a6f (table test-table [id=6f81770057d74f539a49fcfa9a711cfb]) successfully deleted
I20260621 07:04:00.144290 3505 catalog_manager.cc:4841] TS abc28f76126145baa703162ff44c54db (127.1.188.194:39863): tablet 2193960a635c414d8f46d7e540167a6f (table test-table [id=6f81770057d74f539a49fcfa9a711cfb]) successfully deleted
I20260621 07:04:00.149802 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:00.121332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.1.188.193:43689)
0621 07:04:00.131493 (+ 10161us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:00.131498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:00.131522 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:04:00.131586 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:00.131590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:00.131714 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:00.131870 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:00.133855 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.135172 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.137212 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.137939 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.138071 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:00.143956 (+ 5885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:00.143981 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:00.144007 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:00.144051 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:00.147293 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.147995 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.148004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.148007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.148144 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.148456 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.148459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.148461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.148871 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:00.149102 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:00.149329 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 07:04:00.149522 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9982,"thread_start_us":9814,"threads_started":1}
W20260621 07:04:00.150486 3723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43056
W20260621 07:04:00.151204 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3592: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:00.151476 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3592
I20260621 07:04:00.176642 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3732
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:00.192916 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:00.177503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.1.188.194:39863)
0621 07:04:00.178096 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:00.178101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:00.178120 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:04:00.178188 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:00.178193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:00.178260 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:00.178365 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:00.179831 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.180688 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.181922 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.182316 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.182418 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:00.186888 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:00.186904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:00.186925 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:00.186968 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:00.189967 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.190644 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.190653 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.190655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.190783 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.191181 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.191189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.191192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.191646 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:00.192019 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:00.192292 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:04:00.192542 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":337,"threads_started":1}
W20260621 07:04:00.193662 3862 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204
W20260621 07:04:00.194368 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:00.194535 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3732
I20260621 07:04:00.219331 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3871
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:00.236429 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:00.220265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.1.188.195:38035)
0621 07:04:00.221020 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:00.221025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:00.221043 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:04:00.221111 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:00.221116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:00.221193 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:00.221291 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:00.222745 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.223704 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.224942 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:00.225358 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:00.225448 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:00.230189 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:00.230210 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:00.230231 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:00.230269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:00.233408 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.234242 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.234251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.234254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.234413 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:00.234744 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:00.234748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:00.234750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:00.235261 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:00.235686 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:00.235814 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 07:04:00.236003 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":365,"threads_started":1}
W20260621 07:04:00.237154 4001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262
W20260621 07:04:00.237854 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:00.238044 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3871
I20260621 07:04:00.261927 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 3489
I20260621 07:04:00.263356 3561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:00.395823 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 3489
2026-06-21T07:04:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8807 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[4184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[4184](info): set up 2 sockets
Jun 21 07:04:00 dist-test-slave-hlq1 krb5kdc[4184](info): commencing operation
krb5kdc: starting...
W20260621 07:04:01.496095 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.009s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:01 dist-test-slave-hlq1 krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:01.703275 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46499
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:46499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:02.029534 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:02.029834 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:02.029923 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:02.037137 4197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:02.037254 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:02.037300 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:02.037372 4197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:02.037416 4197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:02.047271 4197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:46499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:02.048560 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:02.049839 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:02.061537 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:02.061623 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:02.061635 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:02.062624 4197 server_base.cc:1034] running on GCE node
I20260621 07:04:02.063539 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:02.065068 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:02.066315 4197 hybrid_clock.cc:648] HybridClock initialized: now 1782025442066265 us; error 43 us; skew 500 ppm
I20260621 07:04:02.066706 4197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:02 dist-test-slave-hlq1 krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025442, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:02.069952 4197 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:02.071496 4197 webserver.cc:460] Webserver started at http://127.1.188.254:38873/ using document root <none> and password file <none>
I20260621 07:04:02.072167 4197 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:02.072264 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:02.072525 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:02.075480 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "3e8e7565000842be8c33fa95051b5f3c"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "c393f39712aedfe1b16f5b63de6b5e61"
server_key_iv: "925e51f7caeadffcaccbd94aae352e1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.076222 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "3e8e7565000842be8c33fa95051b5f3c"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "c393f39712aedfe1b16f5b63de6b5e61"
server_key_iv: "925e51f7caeadffcaccbd94aae352e1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.081516 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:04:02.085130 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:02.086584 4197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:04:02.086746 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "3e8e7565000842be8c33fa95051b5f3c"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "c393f39712aedfe1b16f5b63de6b5e61"
server_key_iv: "925e51f7caeadffcaccbd94aae352e1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.086915 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:02.101787 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:02.105756 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:02.105984 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:02.124758 4197 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:46499
I20260621 07:04:02.124790 4263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:46499 every 8 connection(s)
I20260621 07:04:02.126327 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:02.130277 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:02.131362 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4197
I20260621 07:04:02.131615 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:02.132444 1779 external_mini_cluster.cc:1428] Setting key e9b9d9bd3884f5cb9b457149f441744b
I20260621 07:04:02.145169 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c: Bootstrap starting.
Jun 21 07:04:02 dist-test-slave-hlq1 krb5kdc[4184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:02.149165 4264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:02.150239 4264 log.cc:824] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:02.154150 4264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c: No bootstrap required, opened a new log
I20260621 07:04:02.159013 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:02.134712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.1.188.254:46499)
0621 07:04:02.135666 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:02.135679 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:02.135732 (+ 53us) server_negotiation.cc:407] Connection header received
0621 07:04:02.137497 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:02.137527 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:02.137985 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:02.138373 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:02.140248 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.141746 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.143641 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.144423 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.144643 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:02.149596 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:02.149605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:02.149621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:02.149662 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:02.154043 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:02.154813 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:02.154823 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:02.154832 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:02.155082 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:02.155495 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:02.155500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:02.155504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:02.156274 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:02.156618 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:02.156971 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 07:04:02.157482 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":477,"threads_started":1}
I20260621 07:04:02.164808 4264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } }
I20260621 07:04:02.165165 4264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:02.165251 4264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8e7565000842be8c33fa95051b5f3c, State: Initialized, Role: FOLLOWER
I20260621 07:04:02.165818 4264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } }
I20260621 07:04:02.166046 4264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:02.166162 4264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:02.166301 4264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:02.168865 4264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } }
I20260621 07:04:02.169356 4264 leader_election.cc:302] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8e7565000842be8c33fa95051b5f3c; no voters:
I20260621 07:04:02.169893 4264 leader_election.cc:288] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:02.170001 4269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:02.170825 4269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [term 1 LEADER]: Becoming Leader. State: Replica: 3e8e7565000842be8c33fa95051b5f3c, State: Running, Role: LEADER
I20260621 07:04:02.171324 4269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } }
I20260621 07:04:02.172209 4264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:02.174312 4271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8e7565000842be8c33fa95051b5f3c. Latest consensus state: current_term: 1 leader_uuid: "3e8e7565000842be8c33fa95051b5f3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } } }
I20260621 07:04:02.174647 4271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:02.174984 4270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e8e7565000842be8c33fa95051b5f3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8e7565000842be8c33fa95051b5f3c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46499 } } }
I20260621 07:04:02.175491 4270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:02.177810 4275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:02.183425 4275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:02.192036 4275 catalog_manager.cc:1347] Generated new cluster ID: 153f487830fd4854a388e24929fbad9a
I20260621 07:04:02.192195 4275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:02.203970 4275 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:02.205369 4275 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:02.212913 4275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e8e7565000842be8c33fa95051b5f3c: Generated new TSK 0
I20260621 07:04:02.213865 4275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:02.269552 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:02.598030 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:02.598359 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:02.598522 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:02.605707 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:02.605866 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:02.606043 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:02.615950 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:02.617312 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:02.618749 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:02.632226 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:02.632289 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:02.632491 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:02.633008 4290 server_base.cc:1034] running on GCE node
I20260621 07:04:02.633550 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:02.634423 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:02.635650 4290 hybrid_clock.cc:648] HybridClock initialized: now 1782025442635587 us; error 55 us; skew 500 ppm
I20260621 07:04:02.635958 4290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:02 dist-test-slave-hlq1 krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025442, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:02.639232 4290 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:02.640758 4290 webserver.cc:460] Webserver started at http://127.1.188.193:42795/ using document root <none> and password file <none>
I20260621 07:04:02.641455 4290 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:02.641587 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:02.641904 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:02.644814 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "a1e59abfe4f54d0fa3542677780bfe2b"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "45fcb3dee719a97633aacea12e957601"
server_key_iv: "178c489ac603a6456c8b26e3935e5003"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.645596 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e59abfe4f54d0fa3542677780bfe2b"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "45fcb3dee719a97633aacea12e957601"
server_key_iv: "178c489ac603a6456c8b26e3935e5003"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.650740 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:04:02.654316 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:02.655838 4290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:04:02.656021 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "a1e59abfe4f54d0fa3542677780bfe2b"
format_stamp: "Formatted at 2026-06-21 07:04:02 on dist-test-slave-hlq1"
server_key: "45fcb3dee719a97633aacea12e957601"
server_key_iv: "178c489ac603a6456c8b26e3935e5003"
server_key_version: "encryptionkey@0"
I20260621 07:04:02.656217 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:02.675848 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:02.680099 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:02.680390 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:02.681304 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:02.682593 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:02.682673 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:02.682834 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:02.682894 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:02.716715 4290 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33329
I20260621 07:04:02.716801 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33329 every 8 connection(s)
I20260621 07:04:02.717964 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:02.719940 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4290
I20260621 07:04:02.720217 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:02.720947 1779 external_mini_cluster.cc:1428] Setting key 6fd699f4cd33835c1980e48b04bf5c2b
Jun 21 07:04:02 dist-test-slave-hlq1 krb5kdc[4184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025442, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:04:02.738441 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:02.720712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:53927 (local address 127.1.188.254:46499)
0621 07:04:02.721212 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:02.721219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:02.722313 (+ 1094us) server_negotiation.cc:407] Connection header received
0621 07:04:02.724175 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:02.724181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:02.724277 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:02.724391 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:02.725869 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.726574 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.728264 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.728664 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.728797 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:02.732515 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:02.732521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:02.732524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:02.732554 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:02.735126 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:02.735946 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:02.735951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:02.735953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:02.736069 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:02.736557 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:02.736566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:02.736568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:02.736834 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:02.736989 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:02.737795 (+ 806us) server_negotiation.cc:299] Negotiation successful
0621 07:04:02.738025 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":231,"threads_started":1}
I20260621 07:04:02.739609 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:02.721331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46499 (local address 127.1.188.193:53927)
0621 07:04:02.722147 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:02.722192 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:02.723895 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:02.724626 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:02.724635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:02.725154 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:02.725701 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:02.725719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.726732 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.726738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:02.728121 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:02.728132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:02.728894 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:02.728903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:02.729133 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:02.729826 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:02.729853 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:02.732296 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:02.735279 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:02.735288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:02.735305 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:02.735811 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:02.736243 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:02.736248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:02.736252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:02.736439 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:02.736963 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:02.736971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:02.737332 (+ 361us) client_negotiation.cc:769] Sending connection context
0621 07:04:02.737673 (+ 341us) client_negotiation.cc:240] Negotiation successful
0621 07:04:02.738220 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
I20260621 07:04:02.741338 4418 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46499
I20260621 07:04:02.741676 4418 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:02.742578 4418 heartbeater.cc:511] Master 127.1.188.254:46499 requested a full tablet report, sending...
I20260621 07:04:02.745060 4229 ts_manager.cc:194] Registered new tserver with Master: a1e59abfe4f54d0fa3542677780bfe2b (127.1.188.193:33329)
I20260621 07:04:02.746531 4229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:53927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:02.762971 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:03.092325 4424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:03.092599 4424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:03.092696 4424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:03.099835 4424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:03.099948 4424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:03.100066 4424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:03.109671 4424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:03.110940 4424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:03.112234 4424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:03.125216 4429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:03.125847 4430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:03.126775 4432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:03.127274 4424 server_base.cc:1034] running on GCE node
I20260621 07:04:03.128067 4424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:03.129221 4424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:03.130482 4424 hybrid_clock.cc:648] HybridClock initialized: now 1782025443130420 us; error 55 us; skew 500 ppm
I20260621 07:04:03.130848 4424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:03 dist-test-slave-hlq1 krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:03.138934 4424 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:03.140727 4424 webserver.cc:460] Webserver started at http://127.1.188.194:33699/ using document root <none> and password file <none>
I20260621 07:04:03.141506 4424 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:03.141636 4424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:03.141942 4424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:03.144835 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "42bb6706a7ce4e4fbf2e0c56bc491863"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "a93b37ff0cd7d820d5005bd82b96c198"
server_key_iv: "f79739e446ccb9754b2b01079e0064c5"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.145577 4424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "42bb6706a7ce4e4fbf2e0c56bc491863"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "a93b37ff0cd7d820d5005bd82b96c198"
server_key_iv: "f79739e446ccb9754b2b01079e0064c5"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.150545 4424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:04:03.154120 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.155468 4424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:04:03.155628 4424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "42bb6706a7ce4e4fbf2e0c56bc491863"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "a93b37ff0cd7d820d5005bd82b96c198"
server_key_iv: "f79739e446ccb9754b2b01079e0064c5"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.155826 4424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:03.171527 4424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:03.175877 4424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:03.176170 4424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:03.177120 4424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:03.178467 4424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:03.178536 4424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.178700 4424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:03.178757 4424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.210804 4424 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:45167
I20260621 07:04:03.210850 4551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:45167 every 8 connection(s)
I20260621 07:04:03.212157 4424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:03.212913 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4424
I20260621 07:04:03.213121 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:03.213696 1779 external_mini_cluster.cc:1428] Setting key 83111dd526fdf20aff2a71f201bcebb2
Jun 21 07:04:03 dist-test-slave-hlq1 krb5kdc[4184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:04:03.236323 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:03.214967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:47785 (local address 127.1.188.254:46499)
0621 07:04:03.215176 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:03.215182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:03.216759 (+ 1577us) server_negotiation.cc:407] Connection header received
0621 07:04:03.219714 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:03.219718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:03.219790 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:03.219879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:03.222188 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.223370 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.224989 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.225766 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.225910 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:03.229820 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:03.229825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:03.229828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:03.229864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:03.232569 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:03.233505 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:03.233509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:03.233511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:03.233667 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:03.234292 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:03.234300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:03.234307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:03.234615 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:03.234727 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:03.235758 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0621 07:04:03.235969 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:04:03.237339 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:03.215672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46499 (local address 127.1.188.194:47785)
0621 07:04:03.216554 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:03.216606 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:03.219361 (+ 2755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:03.220209 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:03.220223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:03.221086 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:03.221973 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:03.222002 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.223553 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.223557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:03.224840 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:03.224852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.225976 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.225981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:03.226182 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:03.226870 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:03.226894 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:03.229564 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:03.232719 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:03.232731 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:03.232751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:03.233362 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:03.233848 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:03.233852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:03.233855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:03.234055 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:03.234737 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:03.234743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:03.235151 (+ 408us) client_negotiation.cc:769] Sending connection context
0621 07:04:03.235522 (+ 371us) client_negotiation.cc:240] Negotiation successful
0621 07:04:03.236071 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":310,"threads_started":1}
I20260621 07:04:03.238945 4552 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46499
I20260621 07:04:03.239270 4552 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:03.240190 4552 heartbeater.cc:511] Master 127.1.188.254:46499 requested a full tablet report, sending...
I20260621 07:04:03.241972 4229 ts_manager.cc:194] Registered new tserver with Master: 42bb6706a7ce4e4fbf2e0c56bc491863 (127.1.188.194:45167)
I20260621 07:04:03.243465 4229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:47785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:03.260780 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:03.583642 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:03.583909 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:03.584066 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:03.591204 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:03.591307 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:03.591502 4557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:03.600930 4557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:03.602144 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:03.603425 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:03.616453 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:03.616371 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:03.616418 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:03.617209 4557 server_base.cc:1034] running on GCE node
I20260621 07:04:03.617820 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:03.618780 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:03.620020 4557 hybrid_clock.cc:648] HybridClock initialized: now 1782025443619967 us; error 44 us; skew 500 ppm
I20260621 07:04:03.620441 4557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:03 dist-test-slave-hlq1 krb5kdc[4184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:03.624068 4557 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:03.625916 4557 webserver.cc:460] Webserver started at http://127.1.188.195:45143/ using document root <none> and password file <none>
I20260621 07:04:03.626900 4557 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:03.627103 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:03.627542 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:03.631242 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "c8cf12672abe44bfb33afa79ed0c9c20"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "b04bc05ce255cd21cae5cb0fd6c703c1"
server_key_iv: "b6671c53e41b9e113dba3c3b3551641e"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.632088 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "c8cf12672abe44bfb33afa79ed0c9c20"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "b04bc05ce255cd21cae5cb0fd6c703c1"
server_key_iv: "b6671c53e41b9e113dba3c3b3551641e"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.637324 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:04:03.640846 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.642342 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:03.642529 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "c8cf12672abe44bfb33afa79ed0c9c20"
format_stamp: "Formatted at 2026-06-21 07:04:03 on dist-test-slave-hlq1"
server_key: "b04bc05ce255cd21cae5cb0fd6c703c1"
server_key_iv: "b6671c53e41b9e113dba3c3b3551641e"
server_key_version: "encryptionkey@0"
I20260621 07:04:03.642750 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:03.679201 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:03.683902 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:03.684243 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:03.685283 4557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:03.686694 4557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:03.686800 4557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.686956 4557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:03.687032 4557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:03.723907 4557 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:41673
I20260621 07:04:03.723933 4684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:41673 every 8 connection(s)
I20260621 07:04:03.725244 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:03.729116 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4557
I20260621 07:04:03.729302 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:03.729755 1779 external_mini_cluster.cc:1428] Setting key 9a61ea76c87fe70be0cfe125fced29eb
Jun 21 07:04:03 dist-test-slave-hlq1 krb5kdc[4184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025443, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:03.744714 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:03.727633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:40949 (local address 127.1.188.254:46499)
0621 07:04:03.727808 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:03.727813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:03.729032 (+ 1219us) server_negotiation.cc:407] Connection header received
0621 07:04:03.730961 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:03.730967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:03.731049 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:03.731168 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:03.732809 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.733555 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.735087 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.735587 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.735714 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:03.739137 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:03.739143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:03.739146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:03.739187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:03.741654 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:03.742349 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:03.742357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:03.742361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:03.742489 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:03.742895 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:03.742902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:03.742905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:03.743280 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:03.743453 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:03.744134 (+ 681us) server_negotiation.cc:299] Negotiation successful
0621 07:04:03.744364 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:04:03.745705 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:03.728149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46499 (local address 127.1.188.195:40949)
0621 07:04:03.728883 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:03.728931 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:03.730675 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:03.731444 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:03.731455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:03.731983 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:03.732644 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:03.732662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.733723 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.733726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:03.734947 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:03.734958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:03.735832 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:03.735839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:03.736064 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:03.736726 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:03.736753 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:03.738921 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:03.741791 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:03.741801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:03.741818 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:03.742232 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:03.742595 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:03.742600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:03.742604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:03.742788 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:03.743422 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:03.743431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:03.743782 (+ 351us) client_negotiation.cc:769] Sending connection context
0621 07:04:03.744052 (+ 270us) client_negotiation.cc:240] Negotiation successful
0621 07:04:03.744583 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260621 07:04:03.746944 4685 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46499
I20260621 07:04:03.747370 4685 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:03.748250 4685 heartbeater.cc:511] Master 127.1.188.254:46499 requested a full tablet report, sending...
I20260621 07:04:03.749527 4418 heartbeater.cc:503] Master 127.1.188.254:46499 was elected leader, sending a full tablet report...
I20260621 07:04:03.749778 4229 ts_manager.cc:194] Registered new tserver with Master: c8cf12672abe44bfb33afa79ed0c9c20 (127.1.188.195:41673)
I20260621 07:04:03.750643 4229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:40949
I20260621 07:04:03.757118 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:04:03.789397 4420 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:03.786706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.1.188.254:46499)
0621 07:04:03.786909 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:03.786914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:03.787393 (+ 479us) server_negotiation.cc:407] Connection header received
0621 07:04:03.787645 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:03.787650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:03.787722 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:03.787839 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:03.789026 (+ 1187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50862: BlockingRecv error: recv got EOF from 127.0.0.1:50862 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:04:03.792974 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4290
W20260621 07:04:03.795996 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4290: Not authorized: Client connection negotiation failed: client connection to 127.1.188.193:33329: server requires authentication, but client does not have Kerberos credentials available
I20260621 07:04:03.796231 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4290
I20260621 07:04:03.812311 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4424
W20260621 07:04:03.816044 4697 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:03.813447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.1.188.194:45167)
0621 07:04:03.814016 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:03.814026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:03.814060 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:04:03.814143 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:03.814152 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:03.814416 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:03.814584 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:03.815695 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50820: BlockingRecv error: recv got EOF from 127.0.0.1:50820 (error 108)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
W20260621 07:04:03.819197 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4424: Not authorized: Client connection negotiation failed: client connection to 127.1.188.194:45167: server requires authentication, but client does not have Kerberos credentials available
I20260621 07:04:03.819360 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4424
I20260621 07:04:03.835008 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4557
W20260621 07:04:03.837904 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4557: Not authorized: Client connection negotiation failed: client connection to 127.1.188.195:41673: server requires authentication, but client does not have Kerberos credentials available
I20260621 07:04:03.838053 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4557
I20260621 07:04:03.853230 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4197
I20260621 07:04:03.854398 4259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:03.958045 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4197
2026-06-21T07:04:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3555 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:04 dist-test-slave-hlq1 krb5kdc[4702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:04 dist-test-slave-hlq1 krb5kdc[4702](info): set up 2 sockets
Jun 21 07:04:04 dist-test-slave-hlq1 krb5kdc[4702](info): commencing operation
krb5kdc: starting...
W20260621 07:04:05.060130 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:05 dist-test-slave-hlq1 krb5kdc[4702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:05.274679 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41679
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:05.643222 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:05.643639 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:05.643821 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:05.651640 4715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:05.651764 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:05.651850 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:05.651961 4715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:05.652034 4715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:05.662436 4715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:05.664119 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:05.665764 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:05.679337 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:05.679286 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:05.679360 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:05.680131 4715 server_base.cc:1034] running on GCE node
I20260621 07:04:05.681102 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:05.682830 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:05.684062 4715 hybrid_clock.cc:648] HybridClock initialized: now 1782025445683999 us; error 57 us; skew 500 ppm
I20260621 07:04:05.684513 4715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:05 dist-test-slave-hlq1 krb5kdc[4702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025445, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:05.688164 4715 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:05.689862 4715 webserver.cc:460] Webserver started at http://127.1.188.254:45511/ using document root <none> and password file <none>
I20260621 07:04:05.690685 4715 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:05.690822 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:05.691208 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:05.694751 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "2b211314f93f4692a0a043795b827ef5"
format_stamp: "Formatted at 2026-06-21 07:04:05 on dist-test-slave-hlq1"
server_key: "0e634b8dba031f12ed18d25f617c830f"
server_key_iv: "af770cfec976071574d2c1cb18e07493"
server_key_version: "encryptionkey@0"
I20260621 07:04:05.695672 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "2b211314f93f4692a0a043795b827ef5"
format_stamp: "Formatted at 2026-06-21 07:04:05 on dist-test-slave-hlq1"
server_key: "0e634b8dba031f12ed18d25f617c830f"
server_key_iv: "af770cfec976071574d2c1cb18e07493"
server_key_version: "encryptionkey@0"
I20260621 07:04:05.701766 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:04:05.705539 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:05.707051 4715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:04:05.707243 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "2b211314f93f4692a0a043795b827ef5"
format_stamp: "Formatted at 2026-06-21 07:04:05 on dist-test-slave-hlq1"
server_key: "0e634b8dba031f12ed18d25f617c830f"
server_key_iv: "af770cfec976071574d2c1cb18e07493"
server_key_version: "encryptionkey@0"
I20260621 07:04:05.707482 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:05.729362 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:05.734491 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:05.734809 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:05.756826 4715 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41679
I20260621 07:04:05.756834 4781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41679 every 8 connection(s)
I20260621 07:04:05.758423 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:05.762749 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:05.766381 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4715
I20260621 07:04:05.766613 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:05.767256 1779 external_mini_cluster.cc:1428] Setting key 244961a790293538c732f8754b56a925
I20260621 07:04:05.776654 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5: Bootstrap starting.
I20260621 07:04:05.780758 4782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:05.782177 4782 log.cc:824] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:04:05 dist-test-slave-hlq1 krb5kdc[4702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:05.785089 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5: No bootstrap required, opened a new log
I20260621 07:04:05.790867 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:05.769406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.1.188.254:41679)
0621 07:04:05.770306 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:05.770320 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:05.770366 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:04:05.772046 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:05.772072 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:05.772532 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:05.772899 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:05.774848 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:05.776319 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:05.778058 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:05.778812 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:05.778955 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:05.783726 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:05.783735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:05.783751 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:05.783791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:05.786764 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:05.787460 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:05.787470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:05.787482 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:05.787669 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:05.788045 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:05.788050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:05.788053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:05.788628 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:05.788944 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:05.789382 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 07:04:05.789792 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":408,"threads_started":1}
I20260621 07:04:05.792930 4782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } }
I20260621 07:04:05.793319 4782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:05.793438 4782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b211314f93f4692a0a043795b827ef5, State: Initialized, Role: FOLLOWER
I20260621 07:04:05.794049 4782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } }
I20260621 07:04:05.794284 4782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:05.794438 4782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:05.794616 4782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:05.794814 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:04:05.797780 4782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } }
I20260621 07:04:05.798381 4782 leader_election.cc:302] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b211314f93f4692a0a043795b827ef5; no voters:
I20260621 07:04:05.799034 4782 leader_election.cc:288] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:05.799202 4787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:05.803524 4787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b211314f93f4692a0a043795b827ef5, State: Running, Role: LEADER
I20260621 07:04:05.804103 4787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } }
I20260621 07:04:05.806510 4782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:05.807449 4788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b211314f93f4692a0a043795b827ef5. Latest consensus state: current_term: 1 leader_uuid: "2b211314f93f4692a0a043795b827ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } } }
I20260621 07:04:05.807875 4788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:05.808461 4789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b211314f93f4692a0a043795b827ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b211314f93f4692a0a043795b827ef5" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41679 } } }
I20260621 07:04:05.808771 4789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:05.814391 4793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:05.821367 4793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:05.838667 4793 catalog_manager.cc:1347] Generated new cluster ID: b0528a4089c74a49814cb97d67dbcecf
I20260621 07:04:05.838793 4793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:05.849231 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:05.840362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.1.188.254:41679)
0621 07:04:05.840563 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:05.840569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:05.841146 (+ 577us) server_negotiation.cc:407] Connection header received
0621 07:04:05.841328 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:05.841334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:05.841416 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:05.841522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:05.842756 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:05.843841 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:05.845665 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:05.846281 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:05.846383 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:05.847635 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:05.847642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:05.847647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:05.848454 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:05.848518 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:05.848547 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:05.848654 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:05.848806 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:04:05.848889 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:04:05.853178 4793 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:05.854961 4793 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:05.862884 4793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b211314f93f4692a0a043795b827ef5: Generated new TSK 0
I20260621 07:04:05.863984 4793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:04:05.879254 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4715
I20260621 07:04:05.880532 4777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:05.970697 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4715
2026-06-21T07:04:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2006 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:06 dist-test-slave-hlq1 krb5kdc[4816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:06 dist-test-slave-hlq1 krb5kdc[4816](info): set up 2 sockets
Jun 21 07:04:06 dist-test-slave-hlq1 krb5kdc[4816](info): commencing operation
krb5kdc: starting...
W20260621 07:04:07.051802 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:07 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:07.238483 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42657
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:07.581935 4829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:07.582293 4829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:07.582463 4829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:07.589789 4829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:07.589895 4829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:07.589982 4829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:07.590077 4829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:07.590153 4829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:07.600214 4829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:07.601570 4829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:07.602877 4829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:07.614521 4835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:07.614557 4834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:07.614492 4837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:07.615590 4829 server_base.cc:1034] running on GCE node
I20260621 07:04:07.616402 4829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:07.617759 4829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:07.618999 4829 hybrid_clock.cc:648] HybridClock initialized: now 1782025447618940 us; error 54 us; skew 500 ppm
I20260621 07:04:07.619330 4829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:07 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025447, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:07.622574 4829 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:07.624413 4829 webserver.cc:460] Webserver started at http://127.1.188.254:39491/ using document root <none> and password file <none>
I20260621 07:04:07.625236 4829 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:07.625362 4829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:07.625666 4829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:07.629676 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "ba7c297e69544d37ba87332ea8067c2c"
format_stamp: "Formatted at 2026-06-21 07:04:07 on dist-test-slave-hlq1"
server_key: "6e699926cb7d986ef4fd9c9c3b902606"
server_key_iv: "82267a47d9110855e630b71f996ef92b"
server_key_version: "encryptionkey@0"
I20260621 07:04:07.630725 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "ba7c297e69544d37ba87332ea8067c2c"
format_stamp: "Formatted at 2026-06-21 07:04:07 on dist-test-slave-hlq1"
server_key: "6e699926cb7d986ef4fd9c9c3b902606"
server_key_iv: "82267a47d9110855e630b71f996ef92b"
server_key_version: "encryptionkey@0"
I20260621 07:04:07.637113 4829 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260621 07:04:07.640722 4844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:07.642011 4829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:07.642189 4829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "ba7c297e69544d37ba87332ea8067c2c"
format_stamp: "Formatted at 2026-06-21 07:04:07 on dist-test-slave-hlq1"
server_key: "6e699926cb7d986ef4fd9c9c3b902606"
server_key_iv: "82267a47d9110855e630b71f996ef92b"
server_key_version: "encryptionkey@0"
I20260621 07:04:07.642405 4829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:07.669040 4829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:07.673442 4829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:07.673744 4829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:07.692010 4829 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42657
I20260621 07:04:07.692044 4895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42657 every 8 connection(s)
I20260621 07:04:07.693408 4829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:07.697201 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:07.697821 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4829
I20260621 07:04:07.698035 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:07.698715 1779 external_mini_cluster.cc:1428] Setting key 4443b30ce157b244ded7b6b611ba0c2c
I20260621 07:04:07.711732 4896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c: Bootstrap starting.
Jun 21 07:04:07 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:07.715354 4896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:07.716642 4896 log.cc:824] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:07.719583 4896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c: No bootstrap required, opened a new log
I20260621 07:04:07.722213 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:07.700821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.1.188.254:42657)
0621 07:04:07.701631 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:07.701646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:07.701708 (+ 62us) server_negotiation.cc:407] Connection header received
0621 07:04:07.704006 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:07.704030 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:07.704438 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:07.704774 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:07.706446 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:07.707881 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:07.709709 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:07.710360 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:07.710487 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:07.715048 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:07.715056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:07.715070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:07.715114 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:07.718218 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:07.718907 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:07.718915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:07.718921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:07.719090 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:07.719488 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:07.719496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:07.719500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:07.720039 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:07.720350 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:07.720785 (+ 435us) server_negotiation.cc:299] Negotiation successful
0621 07:04:07.721188 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":329,"threads_started":1}
I20260621 07:04:07.728117 4896 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } }
I20260621 07:04:07.728564 4896 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:07.728655 4896 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7c297e69544d37ba87332ea8067c2c, State: Initialized, Role: FOLLOWER
I20260621 07:04:07.729183 4896 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } }
I20260621 07:04:07.729391 4896 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:07.729485 4896 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:07.729604 4896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:07.731853 4896 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } }
I20260621 07:04:07.732468 4896 leader_election.cc:302] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7c297e69544d37ba87332ea8067c2c; no voters:
I20260621 07:04:07.733174 4896 leader_election.cc:288] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:07.733336 4901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:07.734031 4901 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [term 1 LEADER]: Becoming Leader. State: Replica: ba7c297e69544d37ba87332ea8067c2c, State: Running, Role: LEADER
I20260621 07:04:07.734560 4901 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } }
I20260621 07:04:07.735380 4896 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:07.738857 4902 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba7c297e69544d37ba87332ea8067c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } } }
I20260621 07:04:07.739207 4902 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:07.738961 4903 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7c297e69544d37ba87332ea8067c2c. Latest consensus state: current_term: 1 leader_uuid: "ba7c297e69544d37ba87332ea8067c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7c297e69544d37ba87332ea8067c2c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42657 } } }
I20260621 07:04:07.739423 4903 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:07.741513 4906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:07.747618 4906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:07.756196 4906 catalog_manager.cc:1347] Generated new cluster ID: e6f9edca236d49f79f01a97ae3d585a8
I20260621 07:04:07.756337 4906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:07.768692 4906 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:07.770121 4906 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:07.783244 4906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7c297e69544d37ba87332ea8067c2c: Generated new TSK 0
I20260621 07:04:07.784125 4906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:07.838994 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:08.169862 4922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:08.170132 4922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:08.170264 4922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:08.177434 4922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:08.177546 4922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:08.177726 4922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:08.187369 4922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:08.188804 4922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:08.190052 4922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:08.202870 4928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:08.203037 4930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:08.203349 4927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:08.203804 4922 server_base.cc:1034] running on GCE node
I20260621 07:04:08.204403 4922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:08.205363 4922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:08.206614 4922 hybrid_clock.cc:648] HybridClock initialized: now 1782025448206556 us; error 45 us; skew 500 ppm
I20260621 07:04:08.206940 4922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:08 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:08.214967 4922 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:08.216696 4922 webserver.cc:460] Webserver started at http://127.1.188.193:46323/ using document root <none> and password file <none>
I20260621 07:04:08.217454 4922 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:08.217597 4922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:08.217900 4922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:08.220978 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "1d8379235b6a49e085b9b1e4f0f3027b"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "3a0d8fbb796edd8182026cdbdef74dcc"
server_key_iv: "89962a161681e7d9f3a9ee0c77cc526d"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.222049 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "1d8379235b6a49e085b9b1e4f0f3027b"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "3a0d8fbb796edd8182026cdbdef74dcc"
server_key_iv: "89962a161681e7d9f3a9ee0c77cc526d"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.227216 4922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260621 07:04:08.230667 4937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.232112 4922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:08.232287 4922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "1d8379235b6a49e085b9b1e4f0f3027b"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "3a0d8fbb796edd8182026cdbdef74dcc"
server_key_iv: "89962a161681e7d9f3a9ee0c77cc526d"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.232476 4922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:08.246304 4922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:08.250514 4922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:08.250819 4922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:08.251842 4922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:08.253130 4922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:08.253211 4922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.253360 4922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:08.253435 4922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.286196 4922 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:46217
I20260621 07:04:08.286278 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:46217 every 8 connection(s)
I20260621 07:04:08.287505 4922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:08.290468 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 4922
I20260621 07:04:08.290676 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:08.291327 1779 external_mini_cluster.cc:1428] Setting key 1027a5915344f7aba82846f1f4dd67e6
Jun 21 07:04:08 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:08.309149 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:08.290064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:36683 (local address 127.1.188.254:42657)
0621 07:04:08.290567 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:08.290575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:08.291743 (+ 1168us) server_negotiation.cc:407] Connection header received
0621 07:04:08.293735 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:08.293741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:08.293838 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:08.293966 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:08.295413 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.296572 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.298134 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.298841 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.298995 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:08.302331 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:08.302341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:08.302348 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:08.302391 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:08.305929 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:08.306866 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:08.306870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:08.306872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:08.306978 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:08.307480 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:08.307483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:08.307485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:08.307737 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:08.307846 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:08.308728 (+ 882us) server_negotiation.cc:299] Negotiation successful
0621 07:04:08.308888 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:04:08.310660 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:08.290674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42657 (local address 127.1.188.193:36683)
0621 07:04:08.291562 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:08.291608 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:08.293448 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:08.294186 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:08.294194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:08.294671 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:08.295223 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:08.295240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.296738 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.296742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:08.297990 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:08.298002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.299075 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.299080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:08.299289 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:08.300011 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:08.300032 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:08.302119 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:08.306056 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:08.306062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:08.306075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:08.306751 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:08.307087 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:08.307094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:08.307099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:08.307359 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:08.307863 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:08.307871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:08.308305 (+ 434us) client_negotiation.cc:769] Sending connection context
0621 07:04:08.308624 (+ 319us) client_negotiation.cc:240] Negotiation successful
0621 07:04:08.309186 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":342,"threads_started":1}
I20260621 07:04:08.312165 5050 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
I20260621 07:04:08.312567 5050 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:08.313463 5050 heartbeater.cc:511] Master 127.1.188.254:42657 requested a full tablet report, sending...
I20260621 07:04:08.315686 4861 ts_manager.cc:194] Registered new tserver with Master: 1d8379235b6a49e085b9b1e4f0f3027b (127.1.188.193:46217)
I20260621 07:04:08.317102 4861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:36683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:08.334009 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:08.659170 5056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:08.659513 5056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:08.659636 5056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:08.666694 5056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:08.666803 5056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:08.666914 5056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:08.676651 5056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:08.677984 5056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:08.679669 5056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:08.692759 5062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:08.693027 5061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:08.693461 5064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:08.693723 5056 server_base.cc:1034] running on GCE node
I20260621 07:04:08.694344 5056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:08.695276 5056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:08.696499 5056 hybrid_clock.cc:648] HybridClock initialized: now 1782025448696446 us; error 45 us; skew 500 ppm
I20260621 07:04:08.696816 5056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:08 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:08.700076 5056 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:08.706609 5056 webserver.cc:460] Webserver started at http://127.1.188.194:34195/ using document root <none> and password file <none>
I20260621 07:04:08.707334 5056 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:08.707525 5056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:08.707846 5056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:08.710855 5056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "e853b0a90fa44c95bad50d1217967c6f"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "df8cb5a2493276eaae4a53c591ca3948"
server_key_iv: "86b09d4b9bbbfc90c903a270708e89e1"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.711750 5056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "e853b0a90fa44c95bad50d1217967c6f"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "df8cb5a2493276eaae4a53c591ca3948"
server_key_iv: "86b09d4b9bbbfc90c903a270708e89e1"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.717254 5056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:04:08.720906 5071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.722323 5056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:08.722561 5056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "e853b0a90fa44c95bad50d1217967c6f"
format_stamp: "Formatted at 2026-06-21 07:04:08 on dist-test-slave-hlq1"
server_key: "df8cb5a2493276eaae4a53c591ca3948"
server_key_iv: "86b09d4b9bbbfc90c903a270708e89e1"
server_key_version: "encryptionkey@0"
I20260621 07:04:08.722800 5056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:08.737743 5056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:08.742406 5056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:08.742723 5056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:08.743770 5056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:08.745106 5056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:08.745178 5056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.745302 5056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:08.745355 5056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:08.778189 5056 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:42399
I20260621 07:04:08.778230 5183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:42399 every 8 connection(s)
I20260621 07:04:08.779534 5056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:08.782519 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5056
I20260621 07:04:08.782670 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:08.783123 1779 external_mini_cluster.cc:1428] Setting key f5a69f8863185cc0846079efbbe01362
Jun 21 07:04:08 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025448, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:08.799381 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:08.782108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:57981 (local address 127.1.188.254:42657)
0621 07:04:08.782342 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:08.782349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:08.783629 (+ 1280us) server_negotiation.cc:407] Connection header received
0621 07:04:08.785621 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:08.785626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:08.785719 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:08.785820 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:08.787249 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.788013 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.789598 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.790018 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.790169 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:08.793811 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:08.793816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:08.793819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:08.793851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:08.796422 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:08.797125 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:08.797132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:08.797136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:08.797260 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:08.797658 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:08.797664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:08.797675 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:08.797966 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:08.798127 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:08.798808 (+ 681us) server_negotiation.cc:299] Negotiation successful
0621 07:04:08.799035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:04:08.800437 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:08.782603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42657 (local address 127.1.188.194:57981)
0621 07:04:08.783476 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:08.783527 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:08.785376 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:08.786044 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:08.786053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:08.786542 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:08.787089 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:08.787106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.788164 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.788174 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:08.789463 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:08.789475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:08.790249 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:08.790256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:08.790481 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:08.791238 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:08.791260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:08.793539 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:08.796576 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:08.796584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:08.796601 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:08.797010 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:08.797364 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:08.797369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:08.797373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:08.797555 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:08.798087 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:08.798096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:08.798451 (+ 355us) client_negotiation.cc:769] Sending connection context
0621 07:04:08.798720 (+ 269us) client_negotiation.cc:240] Negotiation successful
0621 07:04:08.799264 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":332,"threads_started":1}
I20260621 07:04:08.801882 5184 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:04:08.802295 5184 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:08.803035 5184 heartbeater.cc:511] Master 127.1.188.254:42657 requested a full tablet report, sending...
I20260621 07:04:08.804520 4861 ts_manager.cc:194] Registered new tserver with Master: e853b0a90fa44c95bad50d1217967c6f (127.1.188.194:42399)
I20260621 07:04:08.805402 4861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:57981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:08.824692 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:09.145009 5189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:09.145308 5189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:09.145401 5189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:09.152555 5189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:09.152670 5189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:09.152782 5189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:09.162550 5189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:09.163848 5189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:09.165092 5189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:09.178059 5195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:09.178088 5194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:09.178121 5197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:09.178901 5189 server_base.cc:1034] running on GCE node
I20260621 07:04:09.179610 5189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:09.180552 5189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:09.181754 5189 hybrid_clock.cc:648] HybridClock initialized: now 1782025449181710 us; error 37 us; skew 500 ppm
I20260621 07:04:09.182024 5189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:09.185146 5189 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:09.186619 5189 webserver.cc:460] Webserver started at http://127.1.188.195:44377/ using document root <none> and password file <none>
I20260621 07:04:09.187300 5189 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:09.187449 5189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:09.187803 5189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:09.190666 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "0cb58cdc2c4b4aca883286475792de3e"
format_stamp: "Formatted at 2026-06-21 07:04:09 on dist-test-slave-hlq1"
server_key: "73f25bca78c47d817db76869ae11e801"
server_key_iv: "070e2af0574ff2c6737f84f132dc5344"
server_key_version: "encryptionkey@0"
I20260621 07:04:09.191532 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "0cb58cdc2c4b4aca883286475792de3e"
format_stamp: "Formatted at 2026-06-21 07:04:09 on dist-test-slave-hlq1"
server_key: "73f25bca78c47d817db76869ae11e801"
server_key_iv: "070e2af0574ff2c6737f84f132dc5344"
server_key_version: "encryptionkey@0"
I20260621 07:04:09.196647 5189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:04:09.200126 5204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:09.201416 5189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:09.201593 5189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "0cb58cdc2c4b4aca883286475792de3e"
format_stamp: "Formatted at 2026-06-21 07:04:09 on dist-test-slave-hlq1"
server_key: "73f25bca78c47d817db76869ae11e801"
server_key_iv: "070e2af0574ff2c6737f84f132dc5344"
server_key_version: "encryptionkey@0"
I20260621 07:04:09.201799 5189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:09.222631 5189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:09.226895 5189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:09.227185 5189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:09.228189 5189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:09.229542 5189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:09.229620 5189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:09.229766 5189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:09.229817 5189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:09.262825 5189 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:40951
I20260621 07:04:09.262912 5316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:40951 every 8 connection(s)
I20260621 07:04:09.264143 5189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:09.273221 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5189
I20260621 07:04:09.273391 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:09.273833 1779 external_mini_cluster.cc:1428] Setting key 59d871e052ee57ab579d4243843bc22b
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:09.283825 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.266762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:60661 (local address 127.1.188.254:42657)
0621 07:04:09.266960 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:09.266966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:09.268175 (+ 1209us) server_negotiation.cc:407] Connection header received
0621 07:04:09.269923 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:09.269927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:09.270008 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:09.270096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:09.271515 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.272318 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.273943 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.274530 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.274684 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.278524 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:09.278528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:09.278532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:09.278564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:09.281287 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.281917 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.281921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.281923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.282038 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.282386 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.282389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.282391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.282637 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:09.282746 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:09.283360 (+ 614us) server_negotiation.cc:299] Negotiation successful
0621 07:04:09.283568 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:04:09.284724 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.267280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42657 (local address 127.1.188.195:60661)
0621 07:04:09.268024 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:09.268060 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:09.269671 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:09.270258 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:09.270266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:09.270748 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:09.271291 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:09.271312 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.272461 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.272465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:09.273802 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:09.273814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.274768 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.274774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:09.275052 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.275918 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:09.275951 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:09.278336 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:09.281398 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:09.281404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:09.281415 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:09.281818 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:09.282125 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:09.282128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:09.282130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:09.282301 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:09.282748 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:09.282753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:09.283041 (+ 288us) client_negotiation.cc:769] Sending connection context
0621 07:04:09.283265 (+ 224us) client_negotiation.cc:240] Negotiation successful
0621 07:04:09.283683 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":330,"threads_started":1}
I20260621 07:04:09.286229 5317 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42657
I20260621 07:04:09.286526 5317 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:09.287245 5317 heartbeater.cc:511] Master 127.1.188.254:42657 requested a full tablet report, sending...
I20260621 07:04:09.288892 4861 ts_manager.cc:194] Registered new tserver with Master: 0cb58cdc2c4b4aca883286475792de3e (127.1.188.195:40951)
I20260621 07:04:09.289536 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:04:09.289901 4861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:60661
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260621 07:04:09.319885 5050 heartbeater.cc:503] Master 127.1.188.254:42657 was elected leader, sending a full tablet report...
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:09.336347 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.322367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.1.188.254:42657)
0621 07:04:09.322521 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:09.322526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:09.322991 (+ 465us) server_negotiation.cc:407] Connection header received
0621 07:04:09.323188 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:09.323194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:09.323265 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:09.323386 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:09.324695 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.325415 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.326663 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.327087 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.327221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.330998 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:09.331003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:09.331006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:09.331038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:09.333865 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.334426 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.334430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.334433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.334556 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.334917 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.334921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.334923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.335312 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:09.335607 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:09.335786 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 07:04:09.335977 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 07:04:09.336866 4861 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43814
I20260621 07:04:09.338796 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4922
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:09.354396 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.339842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.1.188.193:46217)
0621 07:04:09.340440 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:09.340447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:09.340471 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:04:09.340531 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:09.340537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:09.340807 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:09.340986 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:09.342004 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.343083 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.344351 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.344951 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.345055 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.348843 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:09.348854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:09.348868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:09.348899 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:09.351743 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.352417 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.352424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.352426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.352543 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.352911 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.352917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.352920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.353251 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:09.353547 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:09.353803 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 07:04:09.354052 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":306,"threads_started":1}
W20260621 07:04:09.355674 5045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44840
W20260621 07:04:09.356734 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:09.356927 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4922
I20260621 07:04:09.372815 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5056
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:09.388203 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.373613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.1.188.194:42399)
0621 07:04:09.374188 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:09.374195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:09.374221 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:09.374281 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:09.374288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:09.374496 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:09.374624 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:09.375929 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.376917 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.378213 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.378749 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.378848 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.382991 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:09.383003 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:09.383017 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:09.383048 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:09.385728 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.386309 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.386315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.386318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.386437 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.386826 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.386829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.386831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.387151 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:09.387372 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:09.387719 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 07:04:09.387911 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":287,"threads_started":1}
W20260621 07:04:09.389444 5179 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49086
W20260621 07:04:09.390525 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5056: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:09.390676 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5056
I20260621 07:04:09.405958 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5189
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[4816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025449, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:09.421973 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:09.407044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.1.188.195:40951)
0621 07:04:09.407765 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:09.407772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:09.407803 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:04:09.407862 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:09.407868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:09.408075 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:09.408200 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:09.409484 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.410514 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.411890 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:09.412461 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:09.412597 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:09.416795 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:09.416806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:09.416820 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:09.416857 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:09.419603 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.420192 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.420198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.420200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.420344 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:09.420620 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:09.420623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:09.420625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:09.420937 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:09.421163 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:09.421395 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 07:04:09.421588 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":409,"threads_started":1}
W20260621 07:04:09.423270 5312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35648
W20260621 07:04:09.424419 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5189: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:04:09.424574 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5189
I20260621 07:04:09.439888 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 4829
I20260621 07:04:09.441066 4891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:09.545555 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 4829
2026-06-21T07:04:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3580 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[5334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[5334](info): set up 2 sockets
Jun 21 07:04:09 dist-test-slave-hlq1 krb5kdc[5334](info): commencing operation
krb5kdc: starting...
W20260621 07:04:10.636059 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:10 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:10.835747 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39421
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:11.150918 5347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:11.151298 5347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:11.151439 5347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:11.158700 5347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:11.158805 5347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:11.158851 5347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:11.158902 5347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:11.158946 5347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:11.169549 5347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:39421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:11.170809 5347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:11.172071 5347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:11.184017 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:11.184160 5353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:11.184322 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:11.184808 5347 server_base.cc:1034] running on GCE node
I20260621 07:04:11.185595 5347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:11.187007 5347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:11.188251 5347 hybrid_clock.cc:648] HybridClock initialized: now 1782025451188197 us; error 35 us; skew 500 ppm
I20260621 07:04:11.188695 5347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:11 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025451, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:11.192114 5347 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:11.193704 5347 webserver.cc:460] Webserver started at http://127.1.188.254:34275/ using document root <none> and password file <none>
I20260621 07:04:11.194396 5347 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:11.194504 5347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:11.194796 5347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:11.198000 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9f4003ecd2b84d7aa0520bb50971119d"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "78b5579b69b5d5e7e329d8478f6e6d0c"
server_key_iv: "38571a3216c42b3840c9e7bbfeda582f"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.198851 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9f4003ecd2b84d7aa0520bb50971119d"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "78b5579b69b5d5e7e329d8478f6e6d0c"
server_key_iv: "38571a3216c42b3840c9e7bbfeda582f"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.204243 5347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 07:04:11.207831 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:11.209170 5347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:04:11.209345 5347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9f4003ecd2b84d7aa0520bb50971119d"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "78b5579b69b5d5e7e329d8478f6e6d0c"
server_key_iv: "38571a3216c42b3840c9e7bbfeda582f"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.209549 5347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:11.253963 5347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:11.258126 5347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:11.258430 5347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:11.276322 5347 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:39421
I20260621 07:04:11.276355 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:39421 every 8 connection(s)
I20260621 07:04:11.277680 5347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:11.281277 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:11.285511 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5347
I20260621 07:04:11.285784 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:11.286517 1779 external_mini_cluster.cc:1428] Setting key 529f7db1439fffcdc903f26da5444726
I20260621 07:04:11.292444 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d: Bootstrap starting.
I20260621 07:04:11.296201 5414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:11.297413 5414 log.cc:824] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:11.301017 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d: No bootstrap required, opened a new log
Jun 21 07:04:11 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:11.308761 5414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } }
I20260621 07:04:11.309180 5414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:11.309374 5414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4003ecd2b84d7aa0520bb50971119d, State: Initialized, Role: FOLLOWER
I20260621 07:04:11.310276 5414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } }
I20260621 07:04:11.310528 5414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:11.310665 5414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:11.310868 5414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:11.311995 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:11.288667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.1.188.254:39421)
0621 07:04:11.289503 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:11.289521 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:11.289565 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:04:11.291481 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:11.291504 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:11.291906 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:11.292237 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:11.294043 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.295484 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.297288 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.298388 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.298601 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:11.303125 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:11.303140 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:11.303159 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:11.303195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:11.306652 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:11.307434 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:11.307461 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:11.307472 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:11.307672 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:11.308087 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:11.308093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:11.308099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:11.308668 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:11.309009 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:11.310230 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0621 07:04:11.310749 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":392,"threads_started":1}
I20260621 07:04:11.313426 5414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } }
I20260621 07:04:11.314013 5414 leader_election.cc:302] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f4003ecd2b84d7aa0520bb50971119d; no voters:
I20260621 07:04:11.314890 5414 leader_election.cc:288] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:11.315012 5419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:11.316030 5419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [term 1 LEADER]: Becoming Leader. State: Replica: 9f4003ecd2b84d7aa0520bb50971119d, State: Running, Role: LEADER
I20260621 07:04:11.316583 5419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } }
I20260621 07:04:11.317106 5414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:11.319322 5421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f4003ecd2b84d7aa0520bb50971119d. Latest consensus state: current_term: 1 leader_uuid: "9f4003ecd2b84d7aa0520bb50971119d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } } }
I20260621 07:04:11.319792 5421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:11.319710 5420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f4003ecd2b84d7aa0520bb50971119d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4003ecd2b84d7aa0520bb50971119d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39421 } } }
I20260621 07:04:11.320073 5420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:11.322911 5426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:11.330185 5426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:11.339341 5426 catalog_manager.cc:1347] Generated new cluster ID: 6a64bc593324469783e925bc3ec4c943
I20260621 07:04:11.339490 5426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:11.357163 5426 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:11.358482 5426 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:11.368202 5426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f4003ecd2b84d7aa0520bb50971119d: Generated new TSK 0
I20260621 07:04:11.369012 5426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:11.422667 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:11.750280 5440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:11.750631 5440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:11.750792 5440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:11.758189 5440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:11.758328 5440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:11.758504 5440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:11.768292 5440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:11.769598 5440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:11.770910 5440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:11.786401 5446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:11.787169 5448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:11.786401 5445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:11.787796 5440 server_base.cc:1034] running on GCE node
I20260621 07:04:11.788487 5440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:11.789510 5440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:11.790738 5440 hybrid_clock.cc:648] HybridClock initialized: now 1782025451790676 us; error 58 us; skew 500 ppm
I20260621 07:04:11.791061 5440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:11 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025451, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:11.794234 5440 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:11.795696 5440 webserver.cc:460] Webserver started at http://127.1.188.193:35423/ using document root <none> and password file <none>
I20260621 07:04:11.796375 5440 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:11.796511 5440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:11.796818 5440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:11.799857 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "e7dc5e311776449bb815b443e5f8abf9"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "6272656475aa0d84698848dcd87532f2"
server_key_iv: "d16c5b0c8673866bcf356815e714dacc"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.800632 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "e7dc5e311776449bb815b443e5f8abf9"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "6272656475aa0d84698848dcd87532f2"
server_key_iv: "d16c5b0c8673866bcf356815e714dacc"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.805724 5440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 07:04:11.809295 5455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:11.810578 5440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:04:11.810761 5440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "e7dc5e311776449bb815b443e5f8abf9"
format_stamp: "Formatted at 2026-06-21 07:04:11 on dist-test-slave-hlq1"
server_key: "6272656475aa0d84698848dcd87532f2"
server_key_iv: "d16c5b0c8673866bcf356815e714dacc"
server_key_version: "encryptionkey@0"
I20260621 07:04:11.811024 5440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:11.827577 5440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:11.831578 5440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:11.831817 5440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:11.832698 5440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:11.833937 5440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:11.834012 5440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:11.834091 5440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:11.834136 5440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:11.867599 5440 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:42015
I20260621 07:04:11.867682 5567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:42015 every 8 connection(s)
I20260621 07:04:11.868976 5440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:11.874013 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5440
I20260621 07:04:11.874209 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:11.874819 1779 external_mini_cluster.cc:1428] Setting key 48584f4e5f8027ae43a262f6f25f18d8
Jun 21 07:04:11 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025451, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:11.891507 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:11.871687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:34683 (local address 127.1.188.254:39421)
0621 07:04:11.872154 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:11.872163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:11.872754 (+ 591us) server_negotiation.cc:407] Connection header received
0621 07:04:11.874668 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:11.874673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:11.874764 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:11.874875 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:11.876525 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.877307 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.879903 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.880309 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.880413 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:11.884567 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:11.884576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:11.884584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:11.884631 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:11.888154 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:11.888868 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:11.888875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:11.888881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:11.889028 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:11.889432 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:11.889440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:11.889446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:11.889840 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:11.889994 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:11.890751 (+ 757us) server_negotiation.cc:299] Negotiation successful
0621 07:04:11.891010 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260621 07:04:11.892334 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:11.871912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39421 (local address 127.1.188.193:34683)
0621 07:04:11.872596 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:11.872646 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:11.874411 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:11.875101 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:11.875109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:11.875730 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:11.876343 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:11.876378 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.877453 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.877461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:11.879745 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:11.879764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:11.880553 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:11.880562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:11.880915 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:11.881694 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:11.881751 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:11.884317 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:11.888289 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:11.888296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:11.888310 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:11.888730 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:11.889136 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:11.889139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:11.889142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:11.889317 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:11.889970 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:11.889976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:11.890337 (+ 361us) client_negotiation.cc:769] Sending connection context
0621 07:04:11.890565 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 07:04:11.891057 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260621 07:04:11.894080 5568 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39421
I20260621 07:04:11.894424 5568 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:04:11.895287 5568 heartbeater.cc:511] Master 127.1.188.254:39421 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
I20260621 07:04:11.897451 5379 ts_manager.cc:194] Registered new tserver with Master: e7dc5e311776449bb815b443e5f8abf9 (127.1.188.193:42015)
I20260621 07:04:11.898871 5379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:34683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:11.918076 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:12.236629 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:12.236908 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:12.237003 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:12.244226 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:12.244349 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:12.244462 5574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:12.254278 5574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:12.255575 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:12.256880 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:12.269510 5579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:12.269541 5580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:12.269491 5582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:12.270664 5574 server_base.cc:1034] running on GCE node
I20260621 07:04:12.271275 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:12.272200 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:12.273432 5574 hybrid_clock.cc:648] HybridClock initialized: now 1782025452273390 us; error 47 us; skew 500 ppm
I20260621 07:04:12.273758 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:12.276949 5574 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:12.278487 5574 webserver.cc:460] Webserver started at http://127.1.188.194:37099/ using document root <none> and password file <none>
I20260621 07:04:12.279174 5574 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:12.279318 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:12.279649 5574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:12.282469 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "e454cb97e5c346e8939e2577f48aadea"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "97a403a37ce25b9a7eed0823b17c286e"
server_key_iv: "f0f4ae313a29e4e87fb602a38a173d07"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.283237 5574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "e454cb97e5c346e8939e2577f48aadea"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "97a403a37ce25b9a7eed0823b17c286e"
server_key_iv: "f0f4ae313a29e4e87fb602a38a173d07"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.288659 5574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:04:12.292042 5589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.293308 5574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:12.293480 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "e454cb97e5c346e8939e2577f48aadea"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "97a403a37ce25b9a7eed0823b17c286e"
server_key_iv: "f0f4ae313a29e4e87fb602a38a173d07"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.293680 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:12.319386 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:12.323349 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:12.323683 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:12.324715 5574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:12.326007 5574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:12.326107 5574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.326251 5574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:12.326320 5574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.359627 5574 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:34533
I20260621 07:04:12.359658 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:34533 every 8 connection(s)
I20260621 07:04:12.360885 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:12.368199 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5574
I20260621 07:04:12.368392 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:12.368901 1779 external_mini_cluster.cc:1428] Setting key bd8e298956c871b054c722099b560244
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:12.380920 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.363347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:60753 (local address 127.1.188.254:39421)
0621 07:04:12.363562 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:12.363566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:12.364655 (+ 1089us) server_negotiation.cc:407] Connection header received
0621 07:04:12.366458 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:12.366464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:12.366553 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:12.366664 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:12.368133 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.369067 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.370820 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.371237 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.371367 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.374773 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:12.374778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:12.374781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:12.374813 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:12.377595 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.378278 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.378283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.378291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.378407 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.378952 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.378960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.378966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.379346 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:12.379519 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:12.380218 (+ 699us) server_negotiation.cc:299] Negotiation successful
0621 07:04:12.380479 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 07:04:12.381637 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.363789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39421 (local address 127.1.188.194:60753)
0621 07:04:12.364492 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:12.364530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:12.366200 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:12.366885 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:12.366892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:12.367375 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:12.367968 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:12.367985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.369250 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.369254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:12.370678 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:12.370693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.371482 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.371490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:12.371718 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.372417 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:12.372440 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:12.374545 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:12.377726 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:12.377734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:12.377750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:12.378155 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:12.378519 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:12.378526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:12.378531 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:12.378790 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:12.379499 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:12.379505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:12.379802 (+ 297us) client_negotiation.cc:769] Sending connection context
0621 07:04:12.380063 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 07:04:12.380549 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":296,"threads_started":1}
I20260621 07:04:12.383015 5702 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39421
I20260621 07:04:12.383324 5702 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:12.384029 5702 heartbeater.cc:511] Master 127.1.188.254:39421 requested a full tablet report, sending...
I20260621 07:04:12.385437 5379 ts_manager.cc:194] Registered new tserver with Master: e454cb97e5c346e8939e2577f48aadea (127.1.188.194:34533)
I20260621 07:04:12.386286 5379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:60753
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:12.410471 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:12.726377 5707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:12.726657 5707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:12.726821 5707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:12.733939 5707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:12.734084 5707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:12.734261 5707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:12.743954 5707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:12.745656 5707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:12.746946 5707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:12.760335 5712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:12.760349 5715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:12.760390 5713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:12.761270 5707 server_base.cc:1034] running on GCE node
I20260621 07:04:12.761991 5707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:12.762952 5707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:12.764176 5707 hybrid_clock.cc:648] HybridClock initialized: now 1782025452764131 us; error 53 us; skew 500 ppm
I20260621 07:04:12.764514 5707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:12.768054 5707 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:12.769896 5707 webserver.cc:460] Webserver started at http://127.1.188.195:46283/ using document root <none> and password file <none>
I20260621 07:04:12.770797 5707 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:12.770903 5707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:12.771166 5707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:12.774061 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "7b32e09795a84351976f23eb9be5dc92"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "98fbd4ca67b81b2b55d64b7c7c2a9254"
server_key_iv: "995601b2e34ea69c6bca03525999837e"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.774772 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "7b32e09795a84351976f23eb9be5dc92"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "98fbd4ca67b81b2b55d64b7c7c2a9254"
server_key_iv: "995601b2e34ea69c6bca03525999837e"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.779858 5707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:04:12.783293 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.784647 5707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:12.784801 5707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "7b32e09795a84351976f23eb9be5dc92"
format_stamp: "Formatted at 2026-06-21 07:04:12 on dist-test-slave-hlq1"
server_key: "98fbd4ca67b81b2b55d64b7c7c2a9254"
server_key_iv: "995601b2e34ea69c6bca03525999837e"
server_key_version: "encryptionkey@0"
I20260621 07:04:12.784992 5707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:12.799194 5707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:12.803718 5707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:12.804073 5707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:12.805014 5707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:12.806273 5707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:12.806341 5707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.806421 5707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:12.806468 5707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:12.838657 5707 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:44125
I20260621 07:04:12.838697 5834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:44125 every 8 connection(s)
I20260621 07:04:12.839912 5707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:12.848481 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5707
I20260621 07:04:12.848691 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:12.849108 1779 external_mini_cluster.cc:1428] Setting key b2d1fee04d9231017ffc61565600b87e
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:12.860425 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.842346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:51699 (local address 127.1.188.254:39421)
0621 07:04:12.842520 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:12.842524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:12.843668 (+ 1144us) server_negotiation.cc:407] Connection header received
0621 07:04:12.845376 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:12.845381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:12.845459 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:12.845546 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:12.846902 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.847730 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.849523 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.850064 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.850205 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.855098 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:12.855103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:12.855106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:12.855138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:12.857472 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.858156 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.858166 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.858169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.858293 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.858703 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.858708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.858711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.859000 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:12.859145 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:12.859860 (+ 715us) server_negotiation.cc:299] Negotiation successful
0621 07:04:12.860077 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:04:12.861414 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.842800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39421 (local address 127.1.188.195:51699)
0621 07:04:12.843518 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:12.843558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:12.845128 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:12.845728 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:12.845735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:12.846227 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:12.846749 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:12.846765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.847852 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.847856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:12.849373 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:12.849385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.850320 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.850331 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:12.850570 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.852271 (+ 1701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:12.852311 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:12.854894 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:12.857612 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:12.857620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:12.857637 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:12.858033 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:12.858396 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:12.858402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:12.858405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:12.858591 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:12.859119 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:12.859127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:12.859516 (+ 389us) client_negotiation.cc:769] Sending connection context
0621 07:04:12.859768 (+ 252us) client_negotiation.cc:240] Negotiation successful
0621 07:04:12.860292 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":249,"threads_started":1}
I20260621 07:04:12.862738 5835 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39421
I20260621 07:04:12.863022 5835 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:12.863847 5835 heartbeater.cc:511] Master 127.1.188.254:39421 requested a full tablet report, sending...
I20260621 07:04:12.865373 5379 ts_manager.cc:194] Registered new tserver with Master: 7b32e09795a84351976f23eb9be5dc92 (127.1.188.195:44125)
I20260621 07:04:12.866140 5379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:51699
I20260621 07:04:12.877303 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:04:12.901678 5568 heartbeater.cc:503] Master 127.1.188.254:39421 was elected leader, sending a full tablet report...
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:12.926748 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.911443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.1.188.193:42015)
0621 07:04:12.912060 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:12.912068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:12.912097 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:04:12.912159 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:12.912166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:12.912385 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:12.912520 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:12.913999 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.915177 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.916405 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.916959 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.917063 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.921146 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:12.921159 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:12.921198 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:12.921233 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:12.924158 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.924893 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.924900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.924902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.925026 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.925467 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.925470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.925472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.925794 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:12.926019 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:12.926269 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:04:12.926455 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":310,"threads_started":1}
Jun 21 07:04:12 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:12.944906 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.931200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.1.188.254:39421)
0621 07:04:12.931490 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:12.931499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:12.931657 (+ 158us) server_negotiation.cc:407] Connection header received
0621 07:04:12.931778 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:12.931782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:12.931855 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:12.931935 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:12.933145 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.933973 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.935211 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.935742 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.935846 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.939581 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:12.939586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:12.939589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:12.939628 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:12.942148 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.942739 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.942743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.942745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.942872 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.943344 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.943348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.943350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.943762 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:12.943950 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:12.944325 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 07:04:12.944510 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260621 07:04:12.945570 5379 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32858
I20260621 07:04:12.946782 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5440
I20260621 07:04:12.959918 5842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:12.947867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.1.188.193:42015)
0621 07:04:12.948080 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:12.948087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:12.948287 (+ 200us) server_negotiation.cc:407] Connection header received
0621 07:04:12.948489 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:12.948494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:12.948578 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:12.948692 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:12.949866 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.950557 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.951877 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:12.952319 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:12.952459 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:12.955020 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:12.955027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:12.955034 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:12.955094 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:12.957650 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.958272 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.958275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.958278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.958397 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:12.958695 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:12.958698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:12.958700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:12.958936 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:12.959103 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:12.959348 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:04:12.959577 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:04:12.960551 5563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:13.044719 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5440
I20260621 07:04:13.061508 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5574
Jun 21 07:04:13 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:13.077683 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:13.062440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.1.188.194:34533)
0621 07:04:13.063030 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:13.063037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:13.063064 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:04:13.063124 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:13.063130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:13.063351 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:13.063547 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:13.064669 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:13.065752 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:13.066985 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:13.067696 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:13.068197 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:13.072053 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:13.072064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:13.072078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:13.072111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:13.075021 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:13.075767 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:13.075774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:13.075776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:13.075902 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:13.076332 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:13.076335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:13.076337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:13.076669 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:13.076902 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:13.077206 (+ 304us) server_negotiation.cc:299] Negotiation successful
0621 07:04:13.077396 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":324,"threads_started":1}
I20260621 07:04:13.079035 5697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:13.161710 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5574
I20260621 07:04:13.177662 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5707
Jun 21 07:04:13 dist-test-slave-hlq1 krb5kdc[5334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:13.194756 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:13.178696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.1.188.195:44125)
0621 07:04:13.179363 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:13.179370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:13.179441 (+ 71us) server_negotiation.cc:407] Connection header received
0621 07:04:13.179516 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:13.179522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:13.179759 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:13.179925 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:13.181204 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:13.182429 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:13.183968 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:13.184512 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:13.184655 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:13.189043 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:13.189058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:13.189077 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:13.189119 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:13.192013 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:13.192684 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:13.192693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:13.192699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:13.192828 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:13.193196 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:13.193201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:13.193204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:13.193541 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:13.193868 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:13.194131 (+ 263us) server_negotiation.cc:299] Negotiation successful
0621 07:04:13.194400 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":355,"threads_started":1}
I20260621 07:04:13.196216 5830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:13.280344 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5707
I20260621 07:04:13.296512 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5347
I20260621 07:04:13.297802 5409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:13.399793 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5347
2026-06-21T07:04:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3853 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:13 dist-test-slave-hlq1 krb5kdc[5854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:13 dist-test-slave-hlq1 krb5kdc[5854](info): set up 2 sockets
Jun 21 07:04:13 dist-test-slave-hlq1 krb5kdc[5854](info): commencing operation
krb5kdc: starting...
W20260621 07:04:14.500275 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:14 dist-test-slave-hlq1 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:14.714416 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:44387
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:15.071287 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:15.071610 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:15.071794 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:15.079704 5867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:15.079804 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:15.079905 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:15.079980 5867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:15.080063 5867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:15.090055 5867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:15.091431 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:15.092811 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:15.104905 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:15.104943 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:15.104854 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:15.105903 5867 server_base.cc:1034] running on GCE node
I20260621 07:04:15.106670 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:15.108147 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:15.109359 5867 hybrid_clock.cc:648] HybridClock initialized: now 1782025455109295 us; error 49 us; skew 500 ppm
I20260621 07:04:15.109678 5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:15 dist-test-slave-hlq1 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:15.112821 5867 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:15.113942 5867 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:15.114030 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:15.114316 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:15.117352 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "78f4f4d201e8447fa577da4c0d3644cd"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "05a5a39f3940eaacc319e11175f4b368"
server_key_iv: "48681055175e69f2f90774193940c089"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.118083 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "78f4f4d201e8447fa577da4c0d3644cd"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "05a5a39f3940eaacc319e11175f4b368"
server_key_iv: "48681055175e69f2f90774193940c089"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.123456 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:04:15.127208 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:15.128656 5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:15.128798 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "78f4f4d201e8447fa577da4c0d3644cd"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "05a5a39f3940eaacc319e11175f4b368"
server_key_iv: "48681055175e69f2f90774193940c089"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.128990 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:15.147159 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:15.151647 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:15.151871 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:15.171345 5867 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:44387
I20260621 07:04:15.171366 5932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:44387 every 8 connection(s)
I20260621 07:04:15.172318 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:15.175791 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5867
I20260621 07:04:15.176033 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:15.176297 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:15.176723 1779 external_mini_cluster.cc:1428] Setting key 2f8f89b5136ac086e933cb3b5fde9942
I20260621 07:04:15.190642 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: Bootstrap starting.
Jun 21 07:04:15 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:15.194796 5933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:15.196242 5933 log.cc:824] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:15.199800 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: No bootstrap required, opened a new log
I20260621 07:04:15.201450 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:15.178732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.1.188.254:44387)
0621 07:04:15.179665 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:15.179681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:15.179750 (+ 69us) server_negotiation.cc:407] Connection header received
0621 07:04:15.181585 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:15.181622 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:15.182144 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:15.182612 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:15.184016 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.185917 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.187220 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.188337 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.188527 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:15.192554 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:15.192566 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:15.192587 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:15.192651 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:15.196596 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:15.197543 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:15.197549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:15.197555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:15.197817 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:15.198179 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:15.198183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:15.198186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:15.198908 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:15.199186 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:15.199544 (+ 358us) server_negotiation.cc:299] Negotiation successful
0621 07:04:15.199974 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":396,"threads_started":1}
I20260621 07:04:15.208225 5933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } }
I20260621 07:04:15.208554 5933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:15.208626 5933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f4f4d201e8447fa577da4c0d3644cd, State: Initialized, Role: FOLLOWER
I20260621 07:04:15.209298 5933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } }
I20260621 07:04:15.209537 5933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:15.209694 5933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:15.209867 5933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:15.212054 5933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } }
I20260621 07:04:15.212517 5933 leader_election.cc:302] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f4f4d201e8447fa577da4c0d3644cd; no voters:
I20260621 07:04:15.213054 5933 leader_election.cc:288] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:15.213212 5938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:15.214080 5938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [term 1 LEADER]: Becoming Leader. State: Replica: 78f4f4d201e8447fa577da4c0d3644cd, State: Running, Role: LEADER
I20260621 07:04:15.214776 5938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } }
I20260621 07:04:15.215525 5933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:15.218051 5939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f4f4d201e8447fa577da4c0d3644cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } } }
I20260621 07:04:15.218335 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f4f4d201e8447fa577da4c0d3644cd. Latest consensus state: current_term: 1 leader_uuid: "78f4f4d201e8447fa577da4c0d3644cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f4f4d201e8447fa577da4c0d3644cd" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 44387 } } }
I20260621 07:04:15.218788 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:15.218784 5939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd [sys.catalog]: This master's current role is: LEADER
W20260621 07:04:15.226462 5954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:04:15.226608 5954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:04:15.226739 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:15.228574 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:15.237672 5947 catalog_manager.cc:1347] Generated new cluster ID: 1823bde8c4514237be786fbfaf00ef07
I20260621 07:04:15.237788 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:15.257812 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:15.259200 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:15.268477 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: Generated new TSK 0
I20260621 07:04:15.269690 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:15.329178 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:15.676249 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:15.676555 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:15.676733 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:15.683916 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:15.684052 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:15.684232 5959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:15.693755 5959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:15.695122 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:15.696451 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:15.709568 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:15.709703 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:15.709745 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:15.710569 5959 server_base.cc:1034] running on GCE node
I20260621 07:04:15.711161 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:15.712211 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:15.713449 5959 hybrid_clock.cc:648] HybridClock initialized: now 1782025455713409 us; error 59 us; skew 500 ppm
I20260621 07:04:15.713771 5959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:15 dist-test-slave-hlq1 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:15.717104 5959 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:15.718269 5959 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:15.718396 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:15.718708 5959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:15.721660 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "768adb386a454ff5bb49a0997a55cb0f"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "837cd83f0e31b70f18264e4abb0ba7e4"
server_key_iv: "2d7423fcf1f5d50ee8afd69fd3bcbdb9"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.722450 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "768adb386a454ff5bb49a0997a55cb0f"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "837cd83f0e31b70f18264e4abb0ba7e4"
server_key_iv: "2d7423fcf1f5d50ee8afd69fd3bcbdb9"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.727376 5959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260621 07:04:15.730854 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:15.732230 5959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:04:15.732402 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "768adb386a454ff5bb49a0997a55cb0f"
format_stamp: "Formatted at 2026-06-21 07:04:15 on dist-test-slave-hlq1"
server_key: "837cd83f0e31b70f18264e4abb0ba7e4"
server_key_iv: "2d7423fcf1f5d50ee8afd69fd3bcbdb9"
server_key_version: "encryptionkey@0"
I20260621 07:04:15.732715 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:15.750033 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:15.754436 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:15.754679 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:15.755734 5959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:15.757077 5959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:15.757169 5959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:15.757241 5959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:15.757300 5959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:15.790572 5959 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:39403
I20260621 07:04:15.790618 6085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:39403 every 8 connection(s)
I20260621 07:04:15.791574 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:15.799853 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 5959
I20260621 07:04:15.800068 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:15.800689 1779 external_mini_cluster.cc:1428] Setting key a956f215241b9d25320c646091218dce
Jun 21 07:04:15 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025455, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:15.813306 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:15.793962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:40361 (local address 127.1.188.254:44387)
0621 07:04:15.794424 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:15.794431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:15.795442 (+ 1011us) server_negotiation.cc:407] Connection header received
0621 07:04:15.797241 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:15.797245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:15.797312 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:15.797412 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:15.798860 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.799782 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.801853 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.802326 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.802472 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:15.806476 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:15.806485 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:15.806493 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:15.806539 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:15.809569 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:15.810633 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:15.810637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:15.810640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:15.810749 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:15.811321 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:15.811325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:15.811327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:15.811687 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:15.811807 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:15.812804 (+ 997us) server_negotiation.cc:299] Negotiation successful
0621 07:04:15.812993 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260621 07:04:15.814913 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:15.794527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:44387 (local address 127.1.188.193:40361)
0621 07:04:15.795234 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:15.795306 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:15.796986 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:15.797608 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:15.797616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:15.798144 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:15.798693 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:15.798715 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.800017 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.800023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:15.801717 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:15.801734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:15.802625 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:15.802634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:15.802898 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:15.803697 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:15.803721 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:15.806228 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:15.809723 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:15.809737 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:15.809761 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:15.810505 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:15.810883 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:15.810892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:15.810900 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:15.811179 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:15.811830 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:15.811842 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:15.812339 (+ 497us) client_negotiation.cc:769] Sending connection context
0621 07:04:15.812686 (+ 347us) client_negotiation.cc:240] Negotiation successful
0621 07:04:15.813324 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260621 07:04:15.816848 6086 heartbeater.cc:347] Connected to a master server at 127.1.188.254:44387
I20260621 07:04:15.817185 6086 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:15.817972 6086 heartbeater.cc:511] Master 127.1.188.254:44387 requested a full tablet report, sending...
I20260621 07:04:15.820495 5898 ts_manager.cc:194] Registered new tserver with Master: 768adb386a454ff5bb49a0997a55cb0f (127.1.188.193:39403)
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:04:15.822209 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:40361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:15.846601 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:16.165639 6092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:16.165962 6092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:16.166131 6092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:16.173415 6092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:16.173555 6092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:16.173730 6092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:16.183220 6092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:16.184517 6092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:16.185781 6092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:16.198736 6100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:16.198686 6098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:16.198781 6097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:16.199515 6092 server_base.cc:1034] running on GCE node
I20260621 07:04:16.200054 6092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:16.200915 6092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:16.202155 6092 hybrid_clock.cc:648] HybridClock initialized: now 1782025456202050 us; error 56 us; skew 500 ppm
I20260621 07:04:16.202478 6092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:16 dist-test-slave-hlq1 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:16.205945 6092 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:16.207104 6092 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:16.207209 6092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:16.207566 6092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:16.210637 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "b86e9bc8d4234ffe95708c6f4e2e979f"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "d3538f1a7e9c3f79b1951af5379d341a"
server_key_iv: "0ede66caba22260d2186bd2fb2586ce0"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.211555 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "b86e9bc8d4234ffe95708c6f4e2e979f"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "d3538f1a7e9c3f79b1951af5379d341a"
server_key_iv: "0ede66caba22260d2186bd2fb2586ce0"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.217049 6092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:04:16.220855 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.222312 6092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:16.222514 6092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "b86e9bc8d4234ffe95708c6f4e2e979f"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "d3538f1a7e9c3f79b1951af5379d341a"
server_key_iv: "0ede66caba22260d2186bd2fb2586ce0"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.222770 6092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:16.237357 6092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:16.241557 6092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:16.241865 6092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:16.242841 6092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:16.244325 6092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:16.244459 6092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.244608 6092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:16.244710 6092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.288339 6092 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:40645
I20260621 07:04:16.288374 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:40645 every 8 connection(s)
I20260621 07:04:16.289495 6092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:16.295540 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6092
I20260621 07:04:16.295753 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:16.296301 1779 external_mini_cluster.cc:1428] Setting key f979a53054b615539bbf30df1db71e30
Jun 21 07:04:16 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:16.314980 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.292119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:52489 (local address 127.1.188.254:44387)
0621 07:04:16.292336 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.292342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.294012 (+ 1670us) server_negotiation.cc:407] Connection header received
0621 07:04:16.297271 (+ 3259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.297279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.297364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.297466 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:16.299922 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.300934 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.303532 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.304336 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.304447 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.308903 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:16.308909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:16.308914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:16.308955 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:16.312054 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.312778 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.312784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.312787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.312907 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.313310 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.313315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.313318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.313579 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:16.313734 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.314411 (+ 677us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.314624 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:04:16.316200 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.292769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:44387 (local address 127.1.188.194:52489)
0621 07:04:16.293810 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:16.293868 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:16.296954 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:16.297769 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:16.297787 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:16.298716 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:16.299673 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.299705 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.301123 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.301128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.303340 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.303356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.304777 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.304786 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.305090 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.305952 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:16.305991 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:16.308680 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:16.312192 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:16.312200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:16.312216 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:16.312641 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:16.313011 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:16.313017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:16.313020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:16.313193 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:16.313708 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:16.313715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:16.314047 (+ 332us) client_negotiation.cc:769] Sending connection context
0621 07:04:16.314314 (+ 267us) client_negotiation.cc:240] Negotiation successful
0621 07:04:16.314825 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":425,"threads_started":1}
I20260621 07:04:16.317754 6219 heartbeater.cc:347] Connected to a master server at 127.1.188.254:44387
I20260621 07:04:16.318060 6219 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:16.318782 6219 heartbeater.cc:511] Master 127.1.188.254:44387 requested a full tablet report, sending...
I20260621 07:04:16.320369 5898 ts_manager.cc:194] Registered new tserver with Master: b86e9bc8d4234ffe95708c6f4e2e979f (127.1.188.194:40645)
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:04:16.321482 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:52489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:16.345923 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:16.673599 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:16.673867 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:16.674028 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:16.681018 6224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:16.681119 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:16.681294 6224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:16.690469 6224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:16.691726 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:16.693004 6224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:16.705217 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:16.705350 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:16.705355 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:16.706784 6224 server_base.cc:1034] running on GCE node
I20260621 07:04:16.707454 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:16.708491 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:16.709695 6224 hybrid_clock.cc:648] HybridClock initialized: now 1782025456709636 us; error 67 us; skew 500 ppm
I20260621 07:04:16.710014 6224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:16 dist-test-slave-hlq1 krb5kdc[5854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:16.713186 6224 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:16.714345 6224 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:16.714454 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:16.714773 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:16.717758 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "0b16341375bb4e79aea9fa842da1e50e"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "e75733b51e691089849b838e879e1419"
server_key_iv: "ef71c4920b59becb1e87ae8d25c2ad12"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.718605 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "0b16341375bb4e79aea9fa842da1e50e"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "e75733b51e691089849b838e879e1419"
server_key_iv: "ef71c4920b59becb1e87ae8d25c2ad12"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.723572 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260621 07:04:16.726832 6238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.728241 6224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:16.728418 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "0b16341375bb4e79aea9fa842da1e50e"
format_stamp: "Formatted at 2026-06-21 07:04:16 on dist-test-slave-hlq1"
server_key: "e75733b51e691089849b838e879e1419"
server_key_iv: "ef71c4920b59becb1e87ae8d25c2ad12"
server_key_version: "encryptionkey@0"
I20260621 07:04:16.728600 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:16.740949 6224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:16.745211 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:16.745482 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:16.746404 6224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:16.747779 6224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:16.747891 6224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.748019 6224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:16.748113 6224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:16.781186 6224 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:40885
I20260621 07:04:16.781275 6350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:40885 every 8 connection(s)
I20260621 07:04:16.782070 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:16.784682 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6224
I20260621 07:04:16.784897 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:16.785356 1779 external_mini_cluster.cc:1428] Setting key cd7d199f34433aa3aeb1a9a4adb43e33
Jun 21 07:04:16 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025456, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:16.802035 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.784563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:58319 (local address 127.1.188.254:44387)
0621 07:04:16.784771 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.784777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.786005 (+ 1228us) server_negotiation.cc:407] Connection header received
0621 07:04:16.788435 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.788442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.788525 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.788602 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:16.790000 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.790797 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.792465 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.792881 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.793013 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.796461 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:16.796466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:16.796469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:16.796499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:16.798789 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.799832 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.799836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.799839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.800002 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.800485 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.800489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.800492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.800854 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:16.800963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.801635 (+ 672us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.801795 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:04:16.802946 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.785046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:44387 (local address 127.1.188.195:58319)
0621 07:04:16.785830 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:16.785882 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:16.788128 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:16.788796 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:16.788803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:16.789308 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:16.789835 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.789851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.790951 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.790957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.792326 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.792339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.793095 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.793103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.793387 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.794153 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:16.794175 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:16.796257 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:16.798966 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:16.798972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:16.798982 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:16.799706 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:16.800127 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:16.800132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:16.800135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:16.800364 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:16.800967 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:16.800973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:16.801323 (+ 350us) client_negotiation.cc:769] Sending connection context
0621 07:04:16.801541 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 07:04:16.801902 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":342,"threads_started":1}
I20260621 07:04:16.804355 6351 heartbeater.cc:347] Connected to a master server at 127.1.188.254:44387
I20260621 07:04:16.804662 6351 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:16.805286 6351 heartbeater.cc:511] Master 127.1.188.254:44387 requested a full tablet report, sending...
I20260621 07:04:16.806567 5898 ts_manager.cc:194] Registered new tserver with Master: 0b16341375bb4e79aea9fa842da1e50e (127.1.188.195:40885)
I20260621 07:04:16.807344 5898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:58319
I20260621 07:04:16.812978 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:04:16.826320 6086 heartbeater.cc:503] Master 127.1.188.254:44387 was elected leader, sending a full tablet report...
I20260621 07:04:16.831761 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.817654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.1.188.254:44387)
0621 07:04:16.817821 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.817825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.818361 (+ 536us) server_negotiation.cc:407] Connection header received
0621 07:04:16.818635 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.818640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.818705 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.818815 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:16.820560 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.821337 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.822708 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.823159 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.823307 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.825759 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:16.825764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:16.825767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:16.825800 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:16.828946 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.829655 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.829662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.829666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.829805 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:16.830194 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:16.830202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:16.830206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:16.830512 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:16.830790 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.831145 (+ 355us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.831363 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:04:16.835810 5898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:16.842645 5898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:16.876379 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.853893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39403 (local address 127.0.0.1:42482)
0621 07:04:16.865466 (+ 11573us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:16.865514 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:16.865730 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:16.867306 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:16.867315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:16.867352 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:16.867763 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.867782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.869846 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.869851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.873161 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.873175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.875268 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.875273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.875656 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.875693 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:04:16.875801 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 07:04:16.875974 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11262,"mutex_wait_us":83,"thread_start_us":16529,"threads_started":1}
I20260621 07:04:16.876492 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.854540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.1.188.195:40885)
0621 07:04:16.864589 (+ 10049us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.864602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.866871 (+ 2269us) server_negotiation.cc:407] Connection header received
0621 07:04:16.866973 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.866980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.867202 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.867391 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:16.868375 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.869889 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.873482 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.874990 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.875117 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.875385 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.875926 (+ 541us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.876104 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9810,"thread_start_us":10077,"threads_started":1}
I20260621 07:04:16.876492 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.854179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.1.188.193:39403)
0621 07:04:16.866410 (+ 12231us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.866421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.866452 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:04:16.866553 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.866561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.866836 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.867016 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:16.867985 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.869661 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.873357 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.874992 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.875592 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.875844 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.875924 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.876113 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12047,"thread_start_us":11891,"threads_started":1}
I20260621 07:04:16.877333 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.854203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:40885 (local address 127.0.0.1:59120)
0621 07:04:16.866622 (+ 12419us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:16.866639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:16.866751 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:16.867677 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:16.867683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:16.867719 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:16.868152 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.868165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.870057 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.870062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.873294 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.873312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.875215 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.875225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.875571 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.875691 (+ 120us) client_negotiation.cc:769] Sending connection context
0621 07:04:16.876694 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0621 07:04:16.876836 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12313,"mutex_wait_us":1210,"thread_start_us":10821,"threads_started":1}
I20260621 07:04:16.880587 6021 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef (DEFAULT_TABLE table=test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:16.881649 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.866572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:40645 (local address 127.0.0.1:57700)
0621 07:04:16.871733 (+ 5161us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:16.871750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:16.871863 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:16.874271 (+ 2408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:16.874276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:16.874303 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:16.874696 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.874709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.876740 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.876745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.878848 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:16.878861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.880868 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.880875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:16.881045 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.881083 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:04:16.881194 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:04:16.881327 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5017,"thread_start_us":4893,"threads_started":1}
I20260621 07:04:16.882119 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:16.870567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.1.188.194:40645)
0621 07:04:16.873705 (+ 3138us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:16.873713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:16.873739 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:16.873833 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:16.873841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:16.874053 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:16.874876 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:16.875089 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.876563 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.879018 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:16.880636 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:16.880798 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:16.880996 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:16.881256 (+ 260us) server_negotiation.cc:299] Negotiation successful
0621 07:04:16.881439 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969,"thread_start_us":2857,"threads_started":1}
I20260621 07:04:16.882453 6021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b4785ce2044c3b685c748da6fe9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:16.881985 6286 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef (DEFAULT_TABLE table=test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:16.884181 6286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b4785ce2044c3b685c748da6fe9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:16.885906 6154 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef (DEFAULT_TABLE table=test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:16.887529 6154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b4785ce2044c3b685c748da6fe9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:16.894351 6369 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Bootstrap starting.
I20260621 07:04:16.897363 6369 tablet_bootstrap.cc:654] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:16.898365 6369 log.cc:824] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:16.900207 6371 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Bootstrap starting.
I20260621 07:04:16.901520 6369 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: No bootstrap required, opened a new log
I20260621 07:04:16.901830 6369 ts_tablet_manager.cc:1397] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 07:04:16.903700 6372 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Bootstrap starting.
I20260621 07:04:16.903780 6371 tablet_bootstrap.cc:654] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:16.905033 6371 log.cc:824] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:16.907043 6372 tablet_bootstrap.cc:654] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:16.908344 6372 log.cc:824] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:16.911012 6372 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: No bootstrap required, opened a new log
I20260621 07:04:16.911352 6372 ts_tablet_manager.cc:1397] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260621 07:04:16.913928 6371 tablet_bootstrap.cc:492] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: No bootstrap required, opened a new log
I20260621 07:04:16.914258 6371 ts_tablet_manager.cc:1397] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260621 07:04:16.914096 6369 raft_consensus.cc:348] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.914583 6369 raft_consensus.cc:374] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:16.914726 6369 raft_consensus.cc:729] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768adb386a454ff5bb49a0997a55cb0f, State: Initialized, Role: FOLLOWER
I20260621 07:04:16.915364 6369 consensus_queue.cc:260] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.918468 6369 ts_tablet_manager.cc:1428] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260621 07:04:16.921633 6372 raft_consensus.cc:348] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.922104 6372 raft_consensus.cc:374] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:16.922219 6372 raft_consensus.cc:729] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86e9bc8d4234ffe95708c6f4e2e979f, State: Initialized, Role: FOLLOWER
I20260621 07:04:16.922907 6372 consensus_queue.cc:260] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.924100 6219 heartbeater.cc:503] Master 127.1.188.254:44387 was elected leader, sending a full tablet report...
I20260621 07:04:16.924762 6372 ts_tablet_manager.cc:1428] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260621 07:04:16.924737 6371 raft_consensus.cc:348] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.925141 6371 raft_consensus.cc:374] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:16.925266 6371 raft_consensus.cc:729] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b16341375bb4e79aea9fa842da1e50e, State: Initialized, Role: FOLLOWER
I20260621 07:04:16.925872 6371 consensus_queue.cc:260] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:16.926980 6351 heartbeater.cc:503] Master 127.1.188.254:44387 was elected leader, sending a full tablet report...
I20260621 07:04:16.927891 6371 ts_tablet_manager.cc:1428] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260621 07:04:17.026911 6376 raft_consensus.cc:482] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:17.027369 6376 raft_consensus.cc:504] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:17.029511 6376 leader_election.cc:288] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 768adb386a454ff5bb49a0997a55cb0f (127.1.188.193:39403), 0b16341375bb4e79aea9fa842da1e50e (127.1.188.195:40885)
W20260621 07:04:17.033706 6352 tablet.cc:2369] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:17.038055 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.030060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:42561 (local address 127.1.188.195:40885)
0621 07:04:17.030309 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.030316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.031579 (+ 1263us) server_negotiation.cc:407] Connection header received
0621 07:04:17.031926 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.031932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.032023 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.032094 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:17.032955 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.034048 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.036047 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.036975 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.037157 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.037252 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.037629 (+ 377us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.037765 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 07:04:17.038102 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.030128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:33227 (local address 127.1.188.193:39403)
0621 07:04:17.030335 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.030343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.031241 (+ 898us) server_negotiation.cc:407] Connection header received
0621 07:04:17.031616 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.031624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.031724 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.031833 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:17.032570 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.033805 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.036056 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.037049 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.037244 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.037322 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.037672 (+ 350us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.037822 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:04:17.038249 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.030060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39403 (local address 127.1.188.194:33227)
0621 07:04:17.031092 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:17.031120 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:17.031380 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:17.031896 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:17.031904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:17.031934 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:17.032395 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:17.032406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.034001 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.034006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:17.035897 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:17.035908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.037269 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.037276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:17.037519 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.037542 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:04:17.037641 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:04:17.037765 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":19,"thread_start_us":492,"threads_started":1}
I20260621 07:04:17.038313 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.030438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:40885 (local address 127.1.188.194:42561)
0621 07:04:17.031386 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:17.031450 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:17.031704 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:17.032272 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:17.032275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:17.032295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:17.032767 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:17.032780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.034204 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.034212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:17.035893 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:17.035905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.037222 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.037228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:17.037445 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.037498 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 07:04:17.037601 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:04:17.037738 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":29,"thread_start_us":522,"threads_started":1}
I20260621 07:04:17.039547 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b4785ce2044c3b685c748da6fe9ef" candidate_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b16341375bb4e79aea9fa842da1e50e" is_pre_election: true
I20260621 07:04:17.039548 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b4785ce2044c3b685c748da6fe9ef" candidate_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768adb386a454ff5bb49a0997a55cb0f" is_pre_election: true
I20260621 07:04:17.040120 6041 raft_consensus.cc:2454] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b86e9bc8d4234ffe95708c6f4e2e979f in term 0.
I20260621 07:04:17.040120 6306 raft_consensus.cc:2454] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b86e9bc8d4234ffe95708c6f4e2e979f in term 0.
I20260621 07:04:17.040901 6109 leader_election.cc:302] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b16341375bb4e79aea9fa842da1e50e, 768adb386a454ff5bb49a0997a55cb0f, b86e9bc8d4234ffe95708c6f4e2e979f; no voters:
I20260621 07:04:17.041390 6376 raft_consensus.cc:2781] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260621 07:04:17.041447 6220 tablet.cc:2369] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:17.041569 6376 raft_consensus.cc:482] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:17.041697 6376 raft_consensus.cc:3037] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 0 FOLLOWER]: Advancing to term 1
W20260621 07:04:17.043769 6087 tablet.cc:2369] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:17.044535 6376 raft_consensus.cc:504] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:17.045185 6376 leader_election.cc:288] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [CANDIDATE]: Term 1 election: Requested vote from peers 768adb386a454ff5bb49a0997a55cb0f (127.1.188.193:39403), 0b16341375bb4e79aea9fa842da1e50e (127.1.188.195:40885)
I20260621 07:04:17.045641 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b4785ce2044c3b685c748da6fe9ef" candidate_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "768adb386a454ff5bb49a0997a55cb0f"
I20260621 07:04:17.045717 6306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b4785ce2044c3b685c748da6fe9ef" candidate_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b16341375bb4e79aea9fa842da1e50e"
I20260621 07:04:17.045856 6041 raft_consensus.cc:3037] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:17.045944 6306 raft_consensus.cc:3037] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:17.048321 6041 raft_consensus.cc:2454] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b86e9bc8d4234ffe95708c6f4e2e979f in term 1.
I20260621 07:04:17.048367 6306 raft_consensus.cc:2454] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b86e9bc8d4234ffe95708c6f4e2e979f in term 1.
I20260621 07:04:17.048772 6107 leader_election.cc:302] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 768adb386a454ff5bb49a0997a55cb0f, b86e9bc8d4234ffe95708c6f4e2e979f; no voters:
I20260621 07:04:17.049067 6376 raft_consensus.cc:2781] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:17.049547 6376 raft_consensus.cc:686] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 1 LEADER]: Becoming Leader. State: Replica: b86e9bc8d4234ffe95708c6f4e2e979f, State: Running, Role: LEADER
I20260621 07:04:17.050081 6376 consensus_queue.cc:237] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } }
I20260621 07:04:17.055627 5898 catalog_manager.cc:5495] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f reported cstate change: term changed from 0 to 1, leader changed from <none> to b86e9bc8d4234ffe95708c6f4e2e979f (127.1.188.194). New cstate: current_term: 1 leader_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86e9bc8d4234ffe95708c6f4e2e979f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40645 } health_report { overall_health: HEALTHY } } }
I20260621 07:04:17.082007 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.076204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.1.188.194:40645)
0621 07:04:17.076451 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.076456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.076577 (+ 121us) server_negotiation.cc:407] Connection header received
0621 07:04:17.076752 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.076756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.076855 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.077013 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:17.077329 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.078105 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.079611 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.080080 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.080189 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.080735 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:17.081387 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:17.081521 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.081663 (+ 142us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.081746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:04:17.092177 6041 raft_consensus.cc:1261] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 1 FOLLOWER]: Refusing update from remote peer b86e9bc8d4234ffe95708c6f4e2e979f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:17.092177 6306 raft_consensus.cc:1261] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 1 FOLLOWER]: Refusing update from remote peer b86e9bc8d4234ffe95708c6f4e2e979f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:17.093291 6376 consensus_queue.cc:1035] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [LEADER]: Connected to new peer: Peer: permanent_uuid: "768adb386a454ff5bb49a0997a55cb0f" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:17.093652 6380 consensus_queue.cc:1035] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b16341375bb4e79aea9fa842da1e50e" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:17.108560 6385 mvcc.cc:205] Tried to move back new op lower bound from 7299176272237006848 to 7299176272081580032. Current Snapshot: MvccSnapshot[applied={T|T < 7299176272237006848}]
I20260621 07:04:17.126847 5897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:17.127238 5897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:17.131544 5897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78f4f4d201e8447fa577da4c0d3644cd: Sending DeleteTablet for 3 replicas of tablet 9f0b4785ce2044c3b685c748da6fe9ef
I20260621 07:04:17.132938 6021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:42482
I20260621 07:04:17.133145 6154 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:57700
I20260621 07:04:17.134214 6393 tablet_replica.cc:330] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: stopping tablet replica
I20260621 07:04:17.134147 6286 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b4785ce2044c3b685c748da6fe9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59120
I20260621 07:04:17.134768 6393 raft_consensus.cc:2229] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:17.135185 6394 tablet_replica.cc:330] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: stopping tablet replica
I20260621 07:04:17.135610 6393 raft_consensus.cc:2258] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:17.135825 6394 raft_consensus.cc:2229] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:17.136461 6394 raft_consensus.cc:2258] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:17.136659 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5959
I20260621 07:04:17.139052 6394 ts_tablet_manager.cc:1905] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:17.143685 6393 ts_tablet_manager.cc:1905] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:17.147006 6394 ts_tablet_manager.cc:1918] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:17.147270 6394 log.cc:1196] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/9f0b4785ce2044c3b685c748da6fe9ef
I20260621 07:04:17.147967 6394 ts_tablet_manager.cc:1939] T 9f0b4785ce2044c3b685c748da6fe9ef P 0b16341375bb4e79aea9fa842da1e50e: Deleting consensus metadata
I20260621 07:04:17.150174 5884 catalog_manager.cc:4841] TS 0b16341375bb4e79aea9fa842da1e50e (127.1.188.195:40885): tablet 9f0b4785ce2044c3b685c748da6fe9ef (table test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]) successfully deleted
I20260621 07:04:17.150929 6392 tablet_replica.cc:330] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: stopping tablet replica
I20260621 07:04:17.151530 6392 raft_consensus.cc:2229] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:17.152148 6392 raft_consensus.cc:2258] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:17.154557 6392 ts_tablet_manager.cc:1905] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:17.157157 6393 ts_tablet_manager.cc:1918] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:17.157420 6393 log.cc:1196] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/9f0b4785ce2044c3b685c748da6fe9ef
I20260621 07:04:17.158175 6393 ts_tablet_manager.cc:1939] T 9f0b4785ce2044c3b685c748da6fe9ef P b86e9bc8d4234ffe95708c6f4e2e979f: Deleting consensus metadata
I20260621 07:04:17.160290 5884 catalog_manager.cc:4841] TS b86e9bc8d4234ffe95708c6f4e2e979f (127.1.188.194:40645): tablet 9f0b4785ce2044c3b685c748da6fe9ef (table test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]) successfully deleted
Jun 21 07:04:17 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:17.162493 6392 ts_tablet_manager.cc:1918] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:17.162684 6392 log.cc:1196] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/9f0b4785ce2044c3b685c748da6fe9ef
I20260621 07:04:17.163297 6392 ts_tablet_manager.cc:1939] T 9f0b4785ce2044c3b685c748da6fe9ef P 768adb386a454ff5bb49a0997a55cb0f: Deleting consensus metadata
I20260621 07:04:17.165634 5882 catalog_manager.cc:4841] TS 768adb386a454ff5bb49a0997a55cb0f (127.1.188.193:39403): tablet 9f0b4785ce2044c3b685c748da6fe9ef (table test-table [id=31e93f5d28a04a6c9c3a1f389a0b534b]) successfully deleted
I20260621 07:04:17.169711 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.139319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.1.188.193:39403)
0621 07:04:17.139597 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.139607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.149198 (+ 9591us) server_negotiation.cc:407] Connection header received
0621 07:04:17.150646 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.150656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.150769 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.150857 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:17.152890 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.154087 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.155922 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.156634 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.156733 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.163027 (+ 6294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:17.163045 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:17.163070 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:17.163120 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:17.166779 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.167647 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.167655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.167657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.167800 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.168229 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.168232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.168234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.168606 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:17.168890 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.169209 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.169385 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260621 07:04:17.170401 6081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:17.269330 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5959
I20260621 07:04:17.295806 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6092
Jun 21 07:04:17 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:17.318078 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.297144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.1.188.194:40645)
0621 07:04:17.297423 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.297431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.297459 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:04:17.297542 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.297550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.297658 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.297801 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:17.299514 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.300987 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.302942 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.303751 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.303916 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.309787 (+ 5871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:17.309803 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:17.309830 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:17.309886 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:17.314512 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.315579 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.315589 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.315592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.315806 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.316421 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.316425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.316427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.316848 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:17.317123 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.317514 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.317710 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 07:04:17.318953 6214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:17.446648 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6092
I20260621 07:04:17.467499 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6224
Jun 21 07:04:17 dist-test-slave-hlq1 krb5kdc[5854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:17.484428 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:17.468376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.1.188.195:40885)
0621 07:04:17.468583 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:17.468592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:17.468653 (+ 61us) server_negotiation.cc:407] Connection header received
0621 07:04:17.468913 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:17.468918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:17.469006 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:17.469127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:17.470580 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.471438 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.472816 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:17.473248 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:17.473389 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:17.478230 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:17.478249 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:17.478271 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:17.478312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:17.481472 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.482172 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.482180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.482183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.482324 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:17.482717 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:17.482723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:17.482727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:17.483147 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:17.483495 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:17.483817 (+ 322us) server_negotiation.cc:299] Negotiation successful
0621 07:04:17.484067 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:04:17.485139 6346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:17.580986 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6224
I20260621 07:04:17.599177 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 5867
I20260621 07:04:17.600579 5928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:17.730309 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 5867
2026-06-21T07:04:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4333 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T07:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:17Z Disabled control of system clock
I20260621 07:04:17.804972 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42295
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:04:18.163586 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:18.163905 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:18.163990 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:18.171851 6403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:18.171976 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:18.172013 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:18.172070 6403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:18.172111 6403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:18.181561 6403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:18.183066 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:18.184960 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:18.197362 6408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:18.197425 6409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:18.197449 6411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:18.200917 6403 server_base.cc:1034] running on GCE node
I20260621 07:04:18.202085 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:18.203765 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:18.205016 6403 hybrid_clock.cc:648] HybridClock initialized: now 1782025458204968 us; error 46 us; skew 500 ppm
I20260621 07:04:18.205412 6403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:18.207912 6403 webserver.cc:460] Webserver started at http://127.1.188.254:41525/ using document root <none> and password file <none>
I20260621 07:04:18.208694 6403 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:18.208832 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:18.209200 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:18.212409 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "ce394ed141754664b4b431e6cbdf6529"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "067581dc73116b910ffaa9fead7dc5df"
server_key_iv: "66d03cf904b3e61bcfa46854ae04fb84"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.213248 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "ce394ed141754664b4b431e6cbdf6529"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "067581dc73116b910ffaa9fead7dc5df"
server_key_iv: "66d03cf904b3e61bcfa46854ae04fb84"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.218751 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:04:18.222589 6417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:18.224100 6403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:18.224314 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "ce394ed141754664b4b431e6cbdf6529"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "067581dc73116b910ffaa9fead7dc5df"
server_key_iv: "66d03cf904b3e61bcfa46854ae04fb84"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.224555 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:18.230505 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:18.231338 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:18.231667 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:18.249837 6403 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42295
I20260621 07:04:18.249867 6468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42295 every 8 connection(s)
I20260621 07:04:18.251293 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:18.255163 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:18.255621 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6403
I20260621 07:04:18.255818 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:18.256471 1779 external_mini_cluster.cc:1428] Setting key 2c5fabf6593b41bb25d083d48757eff5
I20260621 07:04:18.265074 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:18.258503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.1.188.254:42295)
0621 07:04:18.259694 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:18.259705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:18.259750 (+ 45us) server_negotiation.cc:407] Connection header received
0621 07:04:18.261453 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:18.261477 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:18.261928 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:18.262275 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:18.263480 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:18.263487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:18.263500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:18.263515 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:18.263579 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:18.263596 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:18.263776 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:18.263884 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 07:04:18.264152 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":487,"threads_started":1}
I20260621 07:04:18.268229 6469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: Bootstrap starting.
I20260621 07:04:18.271167 6469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:18.272221 6469 log.cc:824] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:18.275031 6469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: No bootstrap required, opened a new log
I20260621 07:04:18.282238 6469 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } }
I20260621 07:04:18.282598 6469 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:18.282677 6469 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce394ed141754664b4b431e6cbdf6529, State: Initialized, Role: FOLLOWER
I20260621 07:04:18.283252 6469 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } }
I20260621 07:04:18.283517 6469 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:18.283641 6469 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:18.283815 6469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:18.286051 6469 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } }
I20260621 07:04:18.286573 6469 leader_election.cc:302] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce394ed141754664b4b431e6cbdf6529; no voters:
I20260621 07:04:18.287174 6469 leader_election.cc:288] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:18.287336 6474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:18.288103 6474 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [term 1 LEADER]: Becoming Leader. State: Replica: ce394ed141754664b4b431e6cbdf6529, State: Running, Role: LEADER
I20260621 07:04:18.288717 6474 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } }
I20260621 07:04:18.289306 6469 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:18.291666 6476 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce394ed141754664b4b431e6cbdf6529. Latest consensus state: current_term: 1 leader_uuid: "ce394ed141754664b4b431e6cbdf6529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } } }
I20260621 07:04:18.291841 6475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce394ed141754664b4b431e6cbdf6529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce394ed141754664b4b431e6cbdf6529" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42295 } } }
I20260621 07:04:18.292125 6476 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:18.292191 6475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:18.293903 6479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:18.299237 6479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:18.308009 6479 catalog_manager.cc:1347] Generated new cluster ID: 2ad4c2cc5881486da4f177639e0467d7
I20260621 07:04:18.308120 6479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:18.333014 6479 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:18.333509 6479 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:18.346025 6479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: Generated new TSK 0
I20260621 07:04:18.346915 6479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:04:18.352474 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--builtin_ntp_servers=127.1.188.212:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 07:04:18.699350 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:18.699659 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:18.699810 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:18.707175 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:18.707289 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:18.707482 6493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:18.716552 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:18.717865 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:18.719674 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:18.733254 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:18.733199 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:18.733229 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:18.734202 6493 server_base.cc:1034] running on GCE node
I20260621 07:04:18.734978 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:18.736111 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:18.737330 6493 hybrid_clock.cc:648] HybridClock initialized: now 1782025458737287 us; error 35 us; skew 500 ppm
I20260621 07:04:18.737694 6493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:18.744364 6493 webserver.cc:460] Webserver started at http://127.1.188.193:41665/ using document root <none> and password file <none>
I20260621 07:04:18.745182 6493 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:18.745319 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:18.745689 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:18.748976 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "3990b125ea9a429a85237a0197458e3d"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "0604afef767556e13913e54c054e33ee"
server_key_iv: "652ac92e258a6f97e00c537eaa6dfa1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.749878 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "3990b125ea9a429a85237a0197458e3d"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "0604afef767556e13913e54c054e33ee"
server_key_iv: "652ac92e258a6f97e00c537eaa6dfa1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.755927 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260621 07:04:18.759714 6507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:18.761234 6493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:18.761477 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "3990b125ea9a429a85237a0197458e3d"
format_stamp: "Formatted at 2026-06-21 07:04:18 on dist-test-slave-hlq1"
server_key: "0604afef767556e13913e54c054e33ee"
server_key_iv: "652ac92e258a6f97e00c537eaa6dfa1e"
server_key_version: "encryptionkey@0"
I20260621 07:04:18.761751 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:18.767616 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:18.768414 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:18.768688 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:18.769634 6493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:18.771083 6493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:18.771169 6493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:18.771323 6493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:18.771441 6493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:18.805084 6493 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:38325
I20260621 07:04:18.805116 6619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:38325 every 8 connection(s)
I20260621 07:04:18.806546 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:18.812922 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6493
I20260621 07:04:18.813138 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:18.813755 1779 external_mini_cluster.cc:1428] Setting key 2c2e85c55c5f7ccb1339cf662f6419c4
I20260621 07:04:18.815326 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:18.809319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:47319 (local address 127.1.188.254:42295)
0621 07:04:18.809774 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:18.809781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:18.810919 (+ 1138us) server_negotiation.cc:407] Connection header received
0621 07:04:18.812691 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:18.812698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:18.812770 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:18.812903 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:18.814489 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:18.814495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:18.814499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:18.814522 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:18.814589 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:18.814610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:18.814732 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:18.814998 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 07:04:18.815062 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260621 07:04:18.816505 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:18.809887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42295 (local address 127.1.188.193:47319)
0621 07:04:18.810771 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:18.810805 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:18.812413 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:18.813208 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:18.813222 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:18.813327 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:18.814040 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:18.814042 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:18.814137 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:18.814222 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:18.814750 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:18.814763 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:18.814775 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 07:04:18.815043 (+ 268us) client_negotiation.cc:240] Negotiation successful
0621 07:04:18.815324 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
I20260621 07:04:18.817231 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--builtin_ntp_servers=127.1.188.212:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:04:18.818034 6620 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42295
I20260621 07:04:18.818363 6620 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:18.818934 6620 heartbeater.cc:511] Master 127.1.188.254:42295 requested a full tablet report, sending...
I20260621 07:04:18.821060 6434 ts_manager.cc:194] Registered new tserver with Master: 3990b125ea9a429a85237a0197458e3d (127.1.188.193:38325)
W20260621 07:04:19.153218 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:19.153609 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:19.153775 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:19.161594 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:19.161731 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:19.161917 6624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:19.171630 6624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:19.173230 6624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:19.175320 6624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:19.189805 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:19.189882 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:19.189888 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:19.196462 6624 server_base.cc:1034] running on GCE node
I20260621 07:04:19.197234 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:19.198450 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:19.199695 6624 hybrid_clock.cc:648] HybridClock initialized: now 1782025459199636 us; error 51 us; skew 500 ppm
I20260621 07:04:19.200062 6624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:19.202703 6624 webserver.cc:460] Webserver started at http://127.1.188.194:38231/ using document root <none> and password file <none>
I20260621 07:04:19.203608 6624 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:19.203743 6624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:19.204144 6624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:19.207572 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "8ea8dd5a7bdf423088cdba024ef693a2"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "336f8de06a4526713484493ab97491aa"
server_key_iv: "5e5c7a4de97b942a7afae61c14477cf7"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.208489 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "8ea8dd5a7bdf423088cdba024ef693a2"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "336f8de06a4526713484493ab97491aa"
server_key_iv: "5e5c7a4de97b942a7afae61c14477cf7"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.214632 6624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260621 07:04:19.218613 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.220185 6624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:19.220376 6624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "8ea8dd5a7bdf423088cdba024ef693a2"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "336f8de06a4526713484493ab97491aa"
server_key_iv: "5e5c7a4de97b942a7afae61c14477cf7"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.220638 6624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:19.227123 6624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:19.228107 6624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:19.228403 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:19.229441 6624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:19.231032 6624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:19.231179 6624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.231304 6624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:19.231391 6624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.273332 6624 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:41379
I20260621 07:04:19.273379 6750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:41379 every 8 connection(s)
I20260621 07:04:19.274859 6624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:19.278610 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6624
I20260621 07:04:19.278784 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:19.279306 1779 external_mini_cluster.cc:1428] Setting key 1945a7ca406f0c5b1eae6310935ebb80
I20260621 07:04:19.282668 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--builtin_ntp_servers=127.1.188.212:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 07:04:19.284864 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.277895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:38811 (local address 127.1.188.254:42295)
0621 07:04:19.278104 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.278110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.280029 (+ 1919us) server_negotiation.cc:407] Connection header received
0621 07:04:19.282122 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.282127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.282217 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.282340 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.283965 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.283975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.283984 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.284011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.284066 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.284088 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.284221 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.284409 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.284487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:04:19.285818 6753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.278455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42295 (local address 127.1.188.194:38811)
0621 07:04:19.279823 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:19.279875 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:19.281819 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:19.282582 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:19.282592 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:19.282705 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:19.283618 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:19.283621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:19.283666 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:19.283762 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:19.284201 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:19.284208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:19.284215 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:04:19.284442 (+ 227us) client_negotiation.cc:240] Negotiation successful
0621 07:04:19.284687 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":491,"threads_started":1}
I20260621 07:04:19.287417 6751 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42295
I20260621 07:04:19.287758 6751 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:19.288414 6751 heartbeater.cc:511] Master 127.1.188.254:42295 requested a full tablet report, sending...
I20260621 07:04:19.290001 6434 ts_manager.cc:194] Registered new tserver with Master: 8ea8dd5a7bdf423088cdba024ef693a2 (127.1.188.194:41379)
W20260621 07:04:19.716594 6754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:19.716912 6754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:19.717010 6754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:19.724783 6754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:19.724895 6754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:19.725023 6754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:19.735195 6754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:19.736871 6754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:19.739034 6754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:19.753437 6760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:19.753818 6759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:19.753881 6762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:19.754482 6754 server_base.cc:1034] running on GCE node
I20260621 07:04:19.755162 6754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:19.756290 6754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:19.757504 6754 hybrid_clock.cc:648] HybridClock initialized: now 1782025459757448 us; error 50 us; skew 500 ppm
I20260621 07:04:19.757874 6754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:19.760579 6754 webserver.cc:460] Webserver started at http://127.1.188.195:35051/ using document root <none> and password file <none>
I20260621 07:04:19.761421 6754 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:19.761541 6754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:19.761920 6754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:19.765431 6754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "dc654d5061f946a88c24b6cb27c08a55"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "c515cd9436c1b94997a365ae77674f25"
server_key_iv: "43b44e9dcd007a7ec84f94209aaec17b"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.766393 6754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "dc654d5061f946a88c24b6cb27c08a55"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "c515cd9436c1b94997a365ae77674f25"
server_key_iv: "43b44e9dcd007a7ec84f94209aaec17b"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.772495 6754 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260621 07:04:19.776636 6768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.778314 6754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:04:19.778537 6754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "dc654d5061f946a88c24b6cb27c08a55"
format_stamp: "Formatted at 2026-06-21 07:04:19 on dist-test-slave-hlq1"
server_key: "c515cd9436c1b94997a365ae77674f25"
server_key_iv: "43b44e9dcd007a7ec84f94209aaec17b"
server_key_version: "encryptionkey@0"
I20260621 07:04:19.778801 6754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:19.785547 6754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:19.786471 6754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:19.786768 6754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:19.787842 6754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:19.789404 6754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:19.789544 6754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.789685 6754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:19.789781 6754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:19.824216 6620 heartbeater.cc:503] Master 127.1.188.254:42295 was elected leader, sending a full tablet report...
I20260621 07:04:19.830410 6754 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:38417
I20260621 07:04:19.830505 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:38417 every 8 connection(s)
I20260621 07:04:19.832067 6754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:19.838829 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6754
I20260621 07:04:19.839025 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:19.839671 1779 external_mini_cluster.cc:1428] Setting key ef3fe7be1ceb9363bd894f845d4d650f
I20260621 07:04:19.842877 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.835442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:57931 (local address 127.1.188.254:42295)
0621 07:04:19.835927 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.835932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.837393 (+ 1461us) server_negotiation.cc:407] Connection header received
0621 07:04:19.839881 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.839886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.839955 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.840058 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.841872 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.841887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.841892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.841921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.841982 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.842004 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.842116 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.842431 (+ 315us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.842503 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260621 07:04:19.844035 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.836167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42295 (local address 127.1.188.195:57931)
0621 07:04:19.837209 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:19.837257 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:19.839497 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:19.840377 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:19.840387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:19.840496 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:19.841453 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:19.841458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:19.841518 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:19.841649 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:19.842136 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:19.842147 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:19.842156 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 07:04:19.842452 (+ 296us) client_negotiation.cc:240] Negotiation successful
0621 07:04:19.842690 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
I20260621 07:04:19.845369 6881 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42295
I20260621 07:04:19.845710 6881 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:19.846381 6881 heartbeater.cc:511] Master 127.1.188.254:42295 requested a full tablet report, sending...
I20260621 07:04:19.847903 6434 ts_manager.cc:194] Registered new tserver with Master: dc654d5061f946a88c24b6cb27c08a55 (127.1.188.195:38417)
I20260621 07:04:19.855726 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:04:19.862799 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.860290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.1.188.254:42295)
0621 07:04:19.860502 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.860511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.861035 (+ 524us) server_negotiation.cc:407] Connection header received
0621 07:04:19.861175 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.861179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.861232 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.861312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.862194 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.862198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.862201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.862216 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.862248 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.862261 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.862355 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.862439 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.862492 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:04:19.867053 6434 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:19.874411 6434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:19.897552 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.889493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:41379 (local address 127.0.0.1:36512)
0621 07:04:19.890760 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:19.890810 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:19.891023 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:19.893187 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:19.893201 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:19.893747 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:19.894917 (+ 1170us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:19.894920 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:19.894975 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:19.895092 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:19.896714 (+ 1622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:19.896722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:19.896730 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:04:19.896895 (+ 165us) client_negotiation.cc:240] Negotiation successful
0621 07:04:19.896962 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":717,"threads_started":1}
I20260621 07:04:19.897866 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.890001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.1.188.194:41379)
0621 07:04:19.892352 (+ 2351us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.892364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.892407 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:04:19.892523 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.892536 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.892892 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.894527 (+ 1635us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.896400 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.896422 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.896440 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.896465 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.896535 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.896560 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.897167 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.897254 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.897472 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":2009,"threads_started":1}
I20260621 07:04:19.898500 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.889643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38417 (local address 127.0.0.1:58096)
0621 07:04:19.892409 (+ 2766us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:19.892430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:19.892548 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:19.894559 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:19.894565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:19.894593 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:19.895362 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:19.895368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:19.895392 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:19.895497 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:19.896190 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:19.896207 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:19.896220 (+ 13us) client_negotiation.cc:769] Sending connection context
0621 07:04:19.896441 (+ 221us) client_negotiation.cc:240] Negotiation successful
0621 07:04:19.896567 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"threads_started":1}
I20260621 07:04:19.899489 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.890017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.1.188.195:38417)
0621 07:04:19.891874 (+ 1857us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.891887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.892669 (+ 782us) server_negotiation.cc:407] Connection header received
0621 07:04:19.892756 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.892764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.893009 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.893208 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.895726 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.895742 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.895765 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.895792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.895897 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.895923 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.898684 (+ 2761us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.898793 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.898919 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":1421,"threads_started":1}
I20260621 07:04:19.903757 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 62ed5b068f32461b929dd82b777695e3 (DEFAULT_TABLE table=test-table [id=81e4b069c865406389e05440171b88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:19.905943 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ed5b068f32461b929dd82b777695e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:19.908154 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.891800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38325 (local address 127.0.0.1:37198)
0621 07:04:19.898122 (+ 6322us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:19.898140 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:19.898209 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:19.905773 (+ 7564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:19.905779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:19.905821 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:19.906532 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:19.906535 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:19.906558 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:19.906621 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:19.907710 (+ 1089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:19.907716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:19.907721 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:04:19.907815 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:04:19.907879 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5914,"thread_start_us":942,"threads_started":1}
I20260621 07:04:19.909037 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:19.892532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.1.188.193:38325)
0621 07:04:19.905166 (+ 12634us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:19.905180 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:19.905222 (+ 42us) server_negotiation.cc:407] Connection header received
0621 07:04:19.905322 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:19.905332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:19.905584 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:19.905784 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:19.907441 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:19.907451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:19.907465 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:19.907484 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:19.907552 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:19.907576 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:19.908584 (+ 1008us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:19.908643 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 07:04:19.908735 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12380,"thread_start_us":12668,"threads_started":1}
I20260621 07:04:19.910112 6816 tablet_service.cc:1467] Processing CreateTablet for tablet 62ed5b068f32461b929dd82b777695e3 (DEFAULT_TABLE table=test-table [id=81e4b069c865406389e05440171b88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:19.912036 6816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ed5b068f32461b929dd82b777695e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:19.912942 6555 tablet_service.cc:1467] Processing CreateTablet for tablet 62ed5b068f32461b929dd82b777695e3 (DEFAULT_TABLE table=test-table [id=81e4b069c865406389e05440171b88f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:19.915072 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62ed5b068f32461b929dd82b777695e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:19.923460 6900 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Bootstrap starting.
I20260621 07:04:19.928236 6900 tablet_bootstrap.cc:654] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:19.928681 6901 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Bootstrap starting.
I20260621 07:04:19.930578 6900 log.cc:824] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:19.933300 6901 tablet_bootstrap.cc:654] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:19.934525 6901 log.cc:824] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:19.937610 6902 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Bootstrap starting.
I20260621 07:04:19.938467 6901 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: No bootstrap required, opened a new log
I20260621 07:04:19.938807 6901 ts_tablet_manager.cc:1397] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260621 07:04:19.938987 6900 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: No bootstrap required, opened a new log
I20260621 07:04:19.939306 6900 ts_tablet_manager.cc:1397] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260621 07:04:19.941164 6902 tablet_bootstrap.cc:654] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:19.942569 6902 log.cc:824] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:19.945612 6902 tablet_bootstrap.cc:492] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: No bootstrap required, opened a new log
I20260621 07:04:19.945956 6902 ts_tablet_manager.cc:1397] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260621 07:04:19.950369 6901 raft_consensus.cc:348] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.950562 6900 raft_consensus.cc:348] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.950906 6901 raft_consensus.cc:374] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:19.950969 6900 raft_consensus.cc:374] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:19.951016 6901 raft_consensus.cc:729] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc654d5061f946a88c24b6cb27c08a55, State: Initialized, Role: FOLLOWER
I20260621 07:04:19.951102 6900 raft_consensus.cc:729] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea8dd5a7bdf423088cdba024ef693a2, State: Initialized, Role: FOLLOWER
I20260621 07:04:19.951807 6901 consensus_queue.cc:260] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.951808 6900 consensus_queue.cc:260] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.953677 6881 heartbeater.cc:503] Master 127.1.188.254:42295 was elected leader, sending a full tablet report...
I20260621 07:04:19.953701 6751 heartbeater.cc:503] Master 127.1.188.254:42295 was elected leader, sending a full tablet report...
I20260621 07:04:19.954006 6900 ts_tablet_manager.cc:1428] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260621 07:04:19.955094 6901 ts_tablet_manager.cc:1428] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260621 07:04:19.954795 6902 raft_consensus.cc:348] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.955241 6902 raft_consensus.cc:374] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:19.955353 6902 raft_consensus.cc:729] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3990b125ea9a429a85237a0197458e3d, State: Initialized, Role: FOLLOWER
I20260621 07:04:19.956202 6902 consensus_queue.cc:260] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:19.958034 6902 ts_tablet_manager.cc:1428] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
W20260621 07:04:20.027284 6752 tablet.cc:2369] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:04:20.059458 6621 tablet.cc:2369] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:04:20.084054 6882 tablet.cc:2369] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:20.122090 6906 raft_consensus.cc:482] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:20.122578 6906 raft_consensus.cc:504] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:20.124806 6906 leader_election.cc:288] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ea8dd5a7bdf423088cdba024ef693a2 (127.1.188.194:41379), 3990b125ea9a429a85237a0197458e3d (127.1.188.193:38325)
I20260621 07:04:20.127833 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.125104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:41379 (local address 127.1.188.195:55467)
0621 07:04:20.125470 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:20.125491 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:20.125626 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:20.126110 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:20.126115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:20.126149 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:20.126967 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:20.126972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:20.126990 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:20.127031 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:20.127348 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:20.127353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:20.127358 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 07:04:20.127495 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 07:04:20.127554 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260621 07:04:20.127856 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.125375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:55467 (local address 127.1.188.194:41379)
0621 07:04:20.125639 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.125649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.125676 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:04:20.125773 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.125783 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.125904 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.126028 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.127184 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.127189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.127192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.127208 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.127238 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.127251 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.127384 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.127494 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.127557 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260621 07:04:20.128444 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.125490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:47203 (local address 127.1.188.193:38325)
0621 07:04:20.125816 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.125825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.126368 (+ 543us) server_negotiation.cc:407] Connection header received
0621 07:04:20.126490 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.126494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.126550 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.126635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.127652 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.127658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.127662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.127689 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.127724 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.127761 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.127897 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.128123 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.128190 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260621 07:04:20.129163 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.125336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38325 (local address 127.1.188.195:47203)
0621 07:04:20.126230 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:20.126248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:20.126372 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:20.126689 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:20.126694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:20.126723 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:20.127390 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:04:20.127430 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:20.127450 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:04:20.127496 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:20.127972 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:20.127979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:20.127986 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:04:20.128691 (+ 705us) client_negotiation.cc:240] Negotiation successful
0621 07:04:20.128744 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":556,"threads_started":1}
I20260621 07:04:20.129195 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ed5b068f32461b929dd82b777695e3" candidate_uuid: "dc654d5061f946a88c24b6cb27c08a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" is_pre_election: true
I20260621 07:04:20.129882 6706 raft_consensus.cc:2454] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc654d5061f946a88c24b6cb27c08a55 in term 0.
I20260621 07:04:20.130446 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ed5b068f32461b929dd82b777695e3" candidate_uuid: "dc654d5061f946a88c24b6cb27c08a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3990b125ea9a429a85237a0197458e3d" is_pre_election: true
I20260621 07:04:20.130630 6769 leader_election.cc:302] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ea8dd5a7bdf423088cdba024ef693a2, dc654d5061f946a88c24b6cb27c08a55; no voters:
I20260621 07:04:20.131055 6906 raft_consensus.cc:2781] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:20.131098 6575 raft_consensus.cc:2454] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc654d5061f946a88c24b6cb27c08a55 in term 0.
I20260621 07:04:20.131229 6906 raft_consensus.cc:482] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:20.131340 6906 raft_consensus.cc:3037] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:20.134052 6906 raft_consensus.cc:504] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:20.134774 6906 leader_election.cc:288] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [CANDIDATE]: Term 1 election: Requested vote from peers 8ea8dd5a7bdf423088cdba024ef693a2 (127.1.188.194:41379), 3990b125ea9a429a85237a0197458e3d (127.1.188.193:38325)
I20260621 07:04:20.135228 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ed5b068f32461b929dd82b777695e3" candidate_uuid: "dc654d5061f946a88c24b6cb27c08a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ea8dd5a7bdf423088cdba024ef693a2"
I20260621 07:04:20.135337 6575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62ed5b068f32461b929dd82b777695e3" candidate_uuid: "dc654d5061f946a88c24b6cb27c08a55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3990b125ea9a429a85237a0197458e3d"
I20260621 07:04:20.135473 6706 raft_consensus.cc:3037] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:20.135635 6575 raft_consensus.cc:3037] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:20.138123 6575 raft_consensus.cc:2454] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc654d5061f946a88c24b6cb27c08a55 in term 1.
I20260621 07:04:20.138135 6706 raft_consensus.cc:2454] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc654d5061f946a88c24b6cb27c08a55 in term 1.
I20260621 07:04:20.138567 6771 leader_election.cc:302] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3990b125ea9a429a85237a0197458e3d, dc654d5061f946a88c24b6cb27c08a55; no voters:
I20260621 07:04:20.138909 6906 raft_consensus.cc:2781] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:20.139472 6906 raft_consensus.cc:686] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 1 LEADER]: Becoming Leader. State: Replica: dc654d5061f946a88c24b6cb27c08a55, State: Running, Role: LEADER
I20260621 07:04:20.140100 6906 consensus_queue.cc:237] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } }
I20260621 07:04:20.145578 6434 catalog_manager.cc:5495] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc654d5061f946a88c24b6cb27c08a55 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "dc654d5061f946a88c24b6cb27c08a55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc654d5061f946a88c24b6cb27c08a55" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260621 07:04:20.200543 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.197928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.1.188.195:38417)
0621 07:04:20.198163 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.198170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.198288 (+ 118us) server_negotiation.cc:407] Connection header received
0621 07:04:20.198598 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.198602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.198704 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.198824 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.199852 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.199856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.199860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.199877 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.199915 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.199927 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.200077 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.200189 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.200260 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:04:20.210955 6706 raft_consensus.cc:1261] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 1 FOLLOWER]: Refusing update from remote peer dc654d5061f946a88c24b6cb27c08a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:20.210955 6575 raft_consensus.cc:1261] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 1 FOLLOWER]: Refusing update from remote peer dc654d5061f946a88c24b6cb27c08a55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:20.212123 6906 consensus_queue.cc:1035] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3990b125ea9a429a85237a0197458e3d" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:20.212466 6910 consensus_queue.cc:1035] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ea8dd5a7bdf423088cdba024ef693a2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:20.226094 6913 mvcc.cc:205] Tried to move back new op lower bound from 7299176285011283968 to 7299176284738588672. Current Snapshot: MvccSnapshot[applied={T|T < 7299176285011283968}]
I20260621 07:04:20.226302 6915 mvcc.cc:205] Tried to move back new op lower bound from 7299176285011283968 to 7299176284738588672. Current Snapshot: MvccSnapshot[applied={T|T < 7299176285011283968}]
I20260621 07:04:20.228142 6914 mvcc.cc:205] Tried to move back new op lower bound from 7299176285011283968 to 7299176284738588672. Current Snapshot: MvccSnapshot[applied={T|T < 7299176285011283968}]
I20260621 07:04:20.242056 6433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:20.242408 6433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:20.246704 6433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce394ed141754664b4b431e6cbdf6529: Sending DeleteTablet for 3 replicas of tablet 62ed5b068f32461b929dd82b777695e3
I20260621 07:04:20.248042 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ed5b068f32461b929dd82b777695e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:20 UTC) from {username='slave'} at 127.0.0.1:36512
I20260621 07:04:20.248143 6555 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ed5b068f32461b929dd82b777695e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:20 UTC) from {username='slave'} at 127.0.0.1:37198
I20260621 07:04:20.248775 6816 tablet_service.cc:1514] Processing DeleteTablet for tablet 62ed5b068f32461b929dd82b777695e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:20 UTC) from {username='slave'} at 127.0.0.1:58096
I20260621 07:04:20.251338 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6493
I20260621 07:04:20.261004 6923 tablet_replica.cc:330] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: stopping tablet replica
I20260621 07:04:20.261806 6923 raft_consensus.cc:2229] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:20.262598 6923 raft_consensus.cc:2258] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:20.263718 6922 tablet_replica.cc:330] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: stopping tablet replica
I20260621 07:04:20.264281 6922 raft_consensus.cc:2229] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:20.265141 6922 raft_consensus.cc:2258] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:20.265506 6923 ts_tablet_manager.cc:1905] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:20.267853 6922 ts_tablet_manager.cc:1905] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:20.273420 6924 tablet_replica.cc:330] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: stopping tablet replica
I20260621 07:04:20.273417 6923 ts_tablet_manager.cc:1918] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:20.273602 6923 log.cc:1196] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/62ed5b068f32461b929dd82b777695e3
I20260621 07:04:20.274163 6923 ts_tablet_manager.cc:1939] T 62ed5b068f32461b929dd82b777695e3 P 3990b125ea9a429a85237a0197458e3d: Deleting consensus metadata
I20260621 07:04:20.274180 6924 raft_consensus.cc:2229] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:20.275094 6924 raft_consensus.cc:2258] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:20.285145 6924 ts_tablet_manager.cc:1905] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:20.286628 6420 catalog_manager.cc:4841] TS 3990b125ea9a429a85237a0197458e3d (127.1.188.193:38325): tablet 62ed5b068f32461b929dd82b777695e3 (table test-table [id=81e4b069c865406389e05440171b88f5]) successfully deleted
I20260621 07:04:20.288479 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.253252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.1.188.193:38325)
0621 07:04:20.253510 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.253520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.283772 (+ 30252us) server_negotiation.cc:407] Connection header received
0621 07:04:20.284251 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.284258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.284407 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.284497 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.287571 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.287578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.287583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.287607 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.287652 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.287670 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.287781 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.288066 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.288145 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:04:20.293398 6924 ts_tablet_manager.cc:1918] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:20.293576 6924 log.cc:1196] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/62ed5b068f32461b929dd82b777695e3
I20260621 07:04:20.293825 6922 ts_tablet_manager.cc:1918] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:20.294013 6922 log.cc:1196] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/62ed5b068f32461b929dd82b777695e3
I20260621 07:04:20.294086 6924 ts_tablet_manager.cc:1939] T 62ed5b068f32461b929dd82b777695e3 P dc654d5061f946a88c24b6cb27c08a55: Deleting consensus metadata
I20260621 07:04:20.294654 6922 ts_tablet_manager.cc:1939] T 62ed5b068f32461b929dd82b777695e3 P 8ea8dd5a7bdf423088cdba024ef693a2: Deleting consensus metadata
I20260621 07:04:20.296089 6419 catalog_manager.cc:4841] TS dc654d5061f946a88c24b6cb27c08a55 (127.1.188.195:38417): tablet 62ed5b068f32461b929dd82b777695e3 (table test-table [id=81e4b069c865406389e05440171b88f5]) successfully deleted
I20260621 07:04:20.296602 6418 catalog_manager.cc:4841] TS 8ea8dd5a7bdf423088cdba024ef693a2 (127.1.188.194:41379): tablet 62ed5b068f32461b929dd82b777695e3 (table test-table [id=81e4b069c865406389e05440171b88f5]) successfully deleted
I20260621 07:04:20.297444 6615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:20.429133 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6493
I20260621 07:04:20.448907 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6624
I20260621 07:04:20.452445 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.449907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.1.188.194:41379)
0621 07:04:20.450144 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.450150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.450176 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:20.450405 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.450410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.450526 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.450645 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.451749 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.451754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.451758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.451775 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.451813 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.451826 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.451967 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.452033 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.452111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 07:04:20.453148 6746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:20.550251 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6624
I20260621 07:04:20.569624 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6754
I20260621 07:04:20.573153 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:20.570594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.1.188.195:38417)
0621 07:04:20.570789 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:20.570798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:20.570900 (+ 102us) server_negotiation.cc:407] Connection header received
0621 07:04:20.571179 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:20.571183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:20.571247 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:20.571380 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:20.572446 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:20.572457 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:20.572460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:20.572476 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:20.572510 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:20.572522 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:20.572673 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:20.572753 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 07:04:20.572821 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:04:20.574131 6876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:20.708839 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6754
I20260621 07:04:20.730156 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6403
I20260621 07:04:20.731801 6464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:20.850366 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6403
2026-06-21T07:04:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3119 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 07:04:20.895905 1779 mini_oidc.cc:152] Starting JWKS server
I20260621 07:04:20.898820 1779 webserver.cc:460] Webserver started at https://127.0.0.1:40799/ using document root <none> and password file <none>
I20260621 07:04:20.899194 1779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:04:20.900010 1779 webserver.cc:460] Webserver started at http://127.0.0.1:40089/ using document root <none> and password file <none>
2026-06-21T07:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:20Z Disabled control of system clock
I20260621 07:04:20.932288 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33189
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:33189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40799/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/testchainca.cert with env {}
W20260621 07:04:21.311884 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:21.312314 6935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:21.312500 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:21.320349 6935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:21.320528 6935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:21.320627 6935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:21.320725 6935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:21.320819 6935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:04:21.320922 6935 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:04:21.321004 6935 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40799/jwks.json
I20260621 07:04:21.331877 6935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:33189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40799/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:21.333611 6935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:21.335299 6935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:21.348717 6941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:21.348654 6940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:21.349283 6943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:21.349814 6935 server_base.cc:1034] running on GCE node
I20260621 07:04:21.350713 6935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:21.352531 6935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:21.353790 6935 hybrid_clock.cc:648] HybridClock initialized: now 1782025461353732 us; error 41 us; skew 500 ppm
I20260621 07:04:21.354337 6935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:21.357157 6935 webserver.cc:460] Webserver started at http://127.1.188.254:44397/ using document root <none> and password file <none>
I20260621 07:04:21.358006 6935 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:21.358108 6935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:21.358486 6935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:21.362972 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "69a9a889675444c4883d0c3888d539a7"
format_stamp: "Formatted at 2026-06-21 07:04:21 on dist-test-slave-hlq1"
server_key: "6a08843853d9ccf88655b64c4551688d"
server_key_iv: "33b9b8b300dee5672cfe135fc26cb833"
server_key_version: "encryptionkey@0"
I20260621 07:04:21.363894 6935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "69a9a889675444c4883d0c3888d539a7"
format_stamp: "Formatted at 2026-06-21 07:04:21 on dist-test-slave-hlq1"
server_key: "6a08843853d9ccf88655b64c4551688d"
server_key_iv: "33b9b8b300dee5672cfe135fc26cb833"
server_key_version: "encryptionkey@0"
I20260621 07:04:21.370020 6935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260621 07:04:21.374058 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:21.375861 6935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:21.376029 6935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "69a9a889675444c4883d0c3888d539a7"
format_stamp: "Formatted at 2026-06-21 07:04:21 on dist-test-slave-hlq1"
server_key: "6a08843853d9ccf88655b64c4551688d"
server_key_iv: "33b9b8b300dee5672cfe135fc26cb833"
server_key_version: "encryptionkey@0"
I20260621 07:04:21.376235 6935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:21.405719 6935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:21.406704 6935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:21.407001 6935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:21.427256 6935 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:33189
I20260621 07:04:21.427302 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:33189 every 8 connection(s)
I20260621 07:04:21.428936 6935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:21.433844 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:21.436419 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 6935
I20260621 07:04:21.436651 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:21.437495 1779 external_mini_cluster.cc:1428] Setting key 4022ae1279f3e6d2ac7f9c666f7b42a7
I20260621 07:04:21.447763 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7: Bootstrap starting.
I20260621 07:04:21.451781 7001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:21.453508 7001 log.cc:824] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:21.455349 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:21.439906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.1.188.254:33189)
0621 07:04:21.441044 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:21.441076 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:21.441129 (+ 53us) server_negotiation.cc:407] Connection header received
0621 07:04:21.443028 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:21.443066 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:21.443948 (+ 882us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:21.444503 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:21.445209 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.446628 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.448275 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.449017 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.449231 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:21.450690 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:21.450698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:21.450714 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:21.451372 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:21.451548 (+ 176us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:21.451575 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:21.451711 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:21.451961 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:04:21.453870 (+ 1909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":386,"threads_started":1}
I20260621 07:04:21.457355 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7: No bootstrap required, opened a new log
I20260621 07:04:21.459966 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:04:21.465919 7001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } }
I20260621 07:04:21.466356 7001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:21.466473 7001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a9a889675444c4883d0c3888d539a7, State: Initialized, Role: FOLLOWER
I20260621 07:04:21.467175 7001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } }
I20260621 07:04:21.467471 7001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:21.467621 7001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:21.467764 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:21.470474 7001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } }
I20260621 07:04:21.471083 7001 leader_election.cc:302] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69a9a889675444c4883d0c3888d539a7; no voters:
I20260621 07:04:21.471810 7001 leader_election.cc:288] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:21.471998 7006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:21.473186 7006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [term 1 LEADER]: Becoming Leader. State: Replica: 69a9a889675444c4883d0c3888d539a7, State: Running, Role: LEADER
I20260621 07:04:21.473778 7006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } }
I20260621 07:04:21.474478 7001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:21.477226 7008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69a9a889675444c4883d0c3888d539a7. Latest consensus state: current_term: 1 leader_uuid: "69a9a889675444c4883d0c3888d539a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } } }
I20260621 07:04:21.477263 7007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69a9a889675444c4883d0c3888d539a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69a9a889675444c4883d0c3888d539a7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33189 } } }
I20260621 07:04:21.477664 7008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:21.477676 7007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:21.479574 7012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:21.486559 7012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:21.497013 7012 catalog_manager.cc:1347] Generated new cluster ID: ef1504adab8247239275c20715e1a2ff
I20260621 07:04:21.497151 7012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:21.515174 7012 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:21.516988 7012 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:21.525142 7012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69a9a889675444c4883d0c3888d539a7: Generated new TSK 0
I20260621 07:04:21.526103 7012 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:04:21.574245 7031 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 07:04:21.574508 7031 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:04:21.575583 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:21.545277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.1.188.254:33189)
0621 07:04:21.545544 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:21.545554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:21.546082 (+ 528us) server_negotiation.cc:407] Connection header received
0621 07:04:21.546448 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:21.546453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:21.546544 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:21.546665 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:04:21.547372 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.548326 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.550553 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.551093 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.551215 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:21.551900 (+ 685us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:04:21.574769 (+ 22869us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:04:21.575177 (+ 408us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260621 07:04:21.575744 7004 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 07:04:21.577706 7030 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:21.545197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33189 (local address 127.0.0.1:38644)
0621 07:04:21.545943 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:21.545976 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:21.546190 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:21.546726 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:21.546732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:21.546747 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:04:21.547187 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:21.547200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.548481 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.548488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:21.550385 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:21.550402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:21.551295 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:21.551301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:21.551516 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:21.551598 (+ 82us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:04:21.575099 (+ 23501us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 07:04:21.575573 (+ 474us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.188.254:33189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":385,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.188.254:33189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 07:04:21.580202 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 6935
I20260621 07:04:21.581585 6996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:21.686326 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 6935
2026-06-21T07:04:21Z chronyd exiting
I20260621 07:04:21.739826 1779 test_util.cc:175] -----------------------------------------------
I20260621 07:04:21.740015 1779 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782025418728693-1779-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (849 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 07:04:21.744338 1779 mini_oidc.cc:152] Starting JWKS server
I20260621 07:04:21.746415 1779 webserver.cc:460] Webserver started at https://127.0.0.1:36587/ using document root <none> and password file <none>
I20260621 07:04:21.746683 1779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:04:21.747493 1779 webserver.cc:460] Webserver started at http://127.0.0.1:36183/ using document root <none> and password file <none>
2026-06-21T07:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:21Z Disabled control of system clock
I20260621 07:04:21.778882 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42739
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42739
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36587/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/testchainca.cert with env {}
W20260621 07:04:22.148078 7038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:22.148438 7038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:22.148625 7038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:22.156461 7038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:22.156581 7038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:22.156673 7038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:22.156780 7038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:22.156869 7038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:04:22.156983 7038 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:04:22.157044 7038 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36587/jwks.json
I20260621 07:04:22.167869 7038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36587/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:22.169672 7038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:22.171314 7038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:22.184696 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:22.184752 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:22.184778 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:22.186129 7038 server_base.cc:1034] running on GCE node
I20260621 07:04:22.186964 7038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:22.188566 7038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:22.189839 7038 hybrid_clock.cc:648] HybridClock initialized: now 1782025462189766 us; error 52 us; skew 500 ppm
I20260621 07:04:22.190271 7038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:22.193001 7038 webserver.cc:460] Webserver started at http://127.1.188.254:42243/ using document root <none> and password file <none>
I20260621 07:04:22.193830 7038 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:22.193979 7038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:22.194365 7038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:22.197775 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3"
format_stamp: "Formatted at 2026-06-21 07:04:22 on dist-test-slave-hlq1"
server_key: "ccb1dceabcce65e0ca5028a8783f7ced"
server_key_iv: "fac50cc78acf2600336ae1b733013797"
server_key_version: "encryptionkey@0"
I20260621 07:04:22.198701 7038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3"
format_stamp: "Formatted at 2026-06-21 07:04:22 on dist-test-slave-hlq1"
server_key: "ccb1dceabcce65e0ca5028a8783f7ced"
server_key_iv: "fac50cc78acf2600336ae1b733013797"
server_key_version: "encryptionkey@0"
I20260621 07:04:22.204567 7038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 07:04:22.208545 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:22.210129 7038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260621 07:04:22.210335 7038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3"
format_stamp: "Formatted at 2026-06-21 07:04:22 on dist-test-slave-hlq1"
server_key: "ccb1dceabcce65e0ca5028a8783f7ced"
server_key_iv: "fac50cc78acf2600336ae1b733013797"
server_key_version: "encryptionkey@0"
I20260621 07:04:22.210589 7038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:22.230058 7038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:22.231032 7038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:22.231388 7038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:22.251884 7038 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42739
I20260621 07:04:22.251927 7103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42739 every 8 connection(s)
I20260621 07:04:22.253535 7038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:22.258128 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:22.260869 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7038
I20260621 07:04:22.261354 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:22.262113 1779 external_mini_cluster.cc:1428] Setting key e69bf6c096e44fcae07a0282521556c7
I20260621 07:04:22.271603 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3: Bootstrap starting.
I20260621 07:04:22.275599 7104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:22.276957 7104 log.cc:824] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:22.277529 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:22.264434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.1.188.254:42739)
0621 07:04:22.265577 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:22.265592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:22.265644 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:04:22.267582 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:22.267614 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:22.268208 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:22.268709 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:22.269385 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.270838 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.272324 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.273068 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.273287 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:22.274719 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:22.274728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:22.274745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:22.275370 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:22.275546 (+ 176us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:22.275571 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:22.275718 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:22.275936 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 07:04:22.276311 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":398,"threads_started":1}
I20260621 07:04:22.280596 7104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3: No bootstrap required, opened a new log
I20260621 07:04:22.281915 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:04:22.288939 7104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } }
I20260621 07:04:22.289400 7104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:22.289525 7104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2f1e736a241b6b284b1fab4f8e0f3, State: Initialized, Role: FOLLOWER
I20260621 07:04:22.290187 7104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } }
I20260621 07:04:22.290438 7104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:22.290607 7104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:22.290778 7104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:22.293503 7104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } }
I20260621 07:04:22.294190 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed2f1e736a241b6b284b1fab4f8e0f3; no voters:
I20260621 07:04:22.294866 7104 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:22.295087 7109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:22.295881 7109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2f1e736a241b6b284b1fab4f8e0f3, State: Running, Role: LEADER
I20260621 07:04:22.296582 7109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } }
I20260621 07:04:22.297506 7104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:22.300211 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed2f1e736a241b6b284b1fab4f8e0f3. Latest consensus state: current_term: 1 leader_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } } }
I20260621 07:04:22.300724 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:22.301553 7110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2f1e736a241b6b284b1fab4f8e0f3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42739 } } }
I20260621 07:04:22.301937 7110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:22.303387 7118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:22.308334 7118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:22.317454 7118 catalog_manager.cc:1347] Generated new cluster ID: e129796c038743ff94d25331451571b4
I20260621 07:04:22.317570 7118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:22.348373 7118 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:22.350198 7118 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:22.359776 7118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed2f1e736a241b6b284b1fab4f8e0f3: Generated new TSK 0
I20260621 07:04:22.361078 7118 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 07:04:22.403501 7134 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 07:04:22.403676 7134 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:04:22.404786 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:22.376244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.1.188.254:42739)
0621 07:04:22.376483 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:22.376492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:22.376898 (+ 406us) server_negotiation.cc:407] Connection header received
0621 07:04:22.377168 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:22.377173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:22.377269 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:22.377435 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:04:22.378045 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.379079 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.381242 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.381724 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.381824 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:22.382391 (+ 567us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:04:22.404178 (+ 21787us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:04:22.404527 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47746: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 07:04:22.404958 7107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47746: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 07:04:22.405207 7133 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:22.376020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42739 (local address 127.0.0.1:47746)
0621 07:04:22.376749 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:22.376769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:22.376955 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:22.377484 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:22.377490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:22.377505 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:04:22.377822 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:22.377836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.379226 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.379232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:22.381100 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:22.381119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:22.381887 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:22.381893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:22.382096 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:22.382149 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:04:22.404449 (+ 22300us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 07:04:22.404762 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.188.254:42739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36587/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":384,"threads_started":1}
I20260621 07:04:22.406899 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7038
I20260621 07:04:22.408360 7099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:22.509042 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7038
2026-06-21T07:04:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (823 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 07:04:22.567771 1779 mini_oidc.cc:152] Starting JWKS server
I20260621 07:04:22.569860 1779 webserver.cc:460] Webserver started at https://127.0.0.1:38685/ using document root <none> and password file <none>
I20260621 07:04:22.570161 1779 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 07:04:22.570891 1779 webserver.cc:460] Webserver started at http://127.0.0.1:35623/ using document root <none> and password file <none>
2026-06-21T07:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:22Z Disabled control of system clock
I20260621 07:04:22.602910 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46211
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:46211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:04:22.970718 7141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:22.971177 7141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:22.971364 7141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:22.978861 7141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:22.978977 7141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:22.979022 7141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:22.979074 7141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:22.979122 7141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:04:22.979188 7141 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:04:22.979229 7141 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38685/jwks.json
I20260621 07:04:22.988888 7141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:46211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:22.990518 7141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:22.992841 7141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:23.006222 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:23.006191 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:23.006260 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:23.007625 7141 server_base.cc:1034] running on GCE node
I20260621 07:04:23.008539 7141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:23.010285 7141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:23.011513 7141 hybrid_clock.cc:648] HybridClock initialized: now 1782025463011467 us; error 32 us; skew 500 ppm
I20260621 07:04:23.011883 7141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:23.014600 7141 webserver.cc:460] Webserver started at http://127.1.188.254:35285/ using document root <none> and password file <none>
I20260621 07:04:23.015499 7141 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:23.015619 7141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:23.016037 7141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:23.019380 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "dcb309ef005a4e068d490c1ff8ab2b8b"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "298aca5462df114bb4be1c66e7becb79"
server_key_iv: "225cc1ef6f8145c70c7f4f2847fb08ef"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.020364 7141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "dcb309ef005a4e068d490c1ff8ab2b8b"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "298aca5462df114bb4be1c66e7becb79"
server_key_iv: "225cc1ef6f8145c70c7f4f2847fb08ef"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.026249 7141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 07:04:23.030125 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:23.031728 7141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:04:23.031921 7141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "dcb309ef005a4e068d490c1ff8ab2b8b"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "298aca5462df114bb4be1c66e7becb79"
server_key_iv: "225cc1ef6f8145c70c7f4f2847fb08ef"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.032146 7141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:23.052636 7141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:23.053529 7141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:23.053951 7141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:23.074190 7141 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:46211
I20260621 07:04:23.074214 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:46211 every 8 connection(s)
I20260621 07:04:23.075763 7141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:23.080052 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:23.083652 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7141
I20260621 07:04:23.083896 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:23.084786 1779 external_mini_cluster.cc:1428] Setting key 03a0e07e48f53b619e94364ccd94e153
I20260621 07:04:23.093881 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b: Bootstrap starting.
I20260621 07:04:23.097749 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:23.099159 7207 log.cc:824] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:23.099977 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:23.087164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.1.188.254:46211)
0621 07:04:23.088288 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:23.088301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:23.088350 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:04:23.090190 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:23.090216 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:23.090802 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:23.091297 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:23.091977 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.093302 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.094783 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.095608 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.095811 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:23.097115 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:23.097124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:23.097140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:23.097756 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:23.097857 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:23.097908 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:23.098029 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:23.098279 (+ 250us) server_negotiation.cc:299] Negotiation successful
0621 07:04:23.098673 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":398,"threads_started":1}
I20260621 07:04:23.102486 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b: No bootstrap required, opened a new log
I20260621 07:04:23.104348 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:04:23.111531 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } }
I20260621 07:04:23.111919 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:23.112037 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb309ef005a4e068d490c1ff8ab2b8b, State: Initialized, Role: FOLLOWER
I20260621 07:04:23.112679 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } }
I20260621 07:04:23.112942 7207 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:23.113098 7207 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:23.113276 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:23.115686 7207 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } }
I20260621 07:04:23.116319 7207 leader_election.cc:302] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb309ef005a4e068d490c1ff8ab2b8b; no voters:
I20260621 07:04:23.116945 7207 leader_election.cc:288] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:23.117120 7212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:23.117882 7212 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [term 1 LEADER]: Becoming Leader. State: Replica: dcb309ef005a4e068d490c1ff8ab2b8b, State: Running, Role: LEADER
I20260621 07:04:23.118556 7212 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } }
I20260621 07:04:23.119509 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:23.122093 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } } }
I20260621 07:04:23.122545 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:23.122605 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb309ef005a4e068d490c1ff8ab2b8b. Latest consensus state: current_term: 1 leader_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb309ef005a4e068d490c1ff8ab2b8b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46211 } } }
I20260621 07:04:23.122946 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:23.126112 7221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:23.130903 7221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:23.140043 7221 catalog_manager.cc:1347] Generated new cluster ID: 40aa0c5cdf344173baebeffc75c145b5
I20260621 07:04:23.140152 7221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:23.153895 7221 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:23.156024 7221 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:23.173269 7221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb309ef005a4e068d490c1ff8ab2b8b: Generated new TSK 0
I20260621 07:04:23.174372 7221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:04:23.204478 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:23.195557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.1.188.254:46211)
0621 07:04:23.195784 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:23.195792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:23.196325 (+ 533us) server_negotiation.cc:407] Connection header received
0621 07:04:23.196571 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:23.196577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:23.196684 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:23.196861 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:04:23.197293 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.198518 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.200042 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.200549 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.200703 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:23.201230 (+ 527us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:04:23.203779 (+ 2549us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:04:23.204137 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 07:04:23.204618 7210 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 07:04:23.206763 7236 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:23.195450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46211 (local address 127.0.0.1:55186)
0621 07:04:23.196215 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:23.196238 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:23.196381 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:23.196871 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:23.196874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:23.196893 (+ 19us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:04:23.197155 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:23.197163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.198682 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.198686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:23.199892 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:23.199907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.200759 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.200769 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:23.200923 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:23.200959 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:04:23.204045 (+ 3086us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 07:04:23.206304 (+ 2259us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.188.254:46211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":1918,"thread_start_us":424,"threads_started":1}
I20260621 07:04:23.207933 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7141
I20260621 07:04:23.209092 7202 generic_service.cc:196] Checking for leaks (request via RPC)
E20260621 07:04:23.217149 7237 webserver.cc:533] Webserver: sslize failed:
E20260621 07:04:23.217254 7237 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 07:04:23.311594 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7141
2026-06-21T07:04:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (799 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T07:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:23Z Disabled control of system clock
I20260621 07:04:23.397863 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:43435
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:43435
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/testchainca.cert with env {}
W20260621 07:04:23.782482 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:23.782845 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:23.782943 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:23.790555 7242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:23.790661 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:23.790701 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:23.790758 7242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:23.790804 7242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 07:04:23.790863 7242 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 07:04:23.790907 7242 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 07:04:23.801443 7242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:43435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:43435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:23.802976 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:23.804494 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:23.817555 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:23.817597 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:23.817636 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:23.818537 7242 server_base.cc:1034] running on GCE node
I20260621 07:04:23.819507 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:23.821183 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:23.822423 7242 hybrid_clock.cc:648] HybridClock initialized: now 1782025463822354 us; error 70 us; skew 500 ppm
I20260621 07:04:23.822804 7242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:23.831326 7242 webserver.cc:460] Webserver started at http://127.1.188.254:46501/ using document root <none> and password file <none>
I20260621 07:04:23.832267 7242 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:23.832427 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:23.832814 7242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:23.836198 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "b80fccbf1d81404abf80086a1f1c0cca"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "19fd4bd2ebbeca84434bd1f869448d89"
server_key_iv: "a0217221de71dfd59cb10debeeee2cb5"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.837111 7242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "b80fccbf1d81404abf80086a1f1c0cca"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "19fd4bd2ebbeca84434bd1f869448d89"
server_key_iv: "a0217221de71dfd59cb10debeeee2cb5"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.843003 7242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 07:04:23.846765 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:23.848271 7242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:23.848481 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "b80fccbf1d81404abf80086a1f1c0cca"
format_stamp: "Formatted at 2026-06-21 07:04:23 on dist-test-slave-hlq1"
server_key: "19fd4bd2ebbeca84434bd1f869448d89"
server_key_iv: "a0217221de71dfd59cb10debeeee2cb5"
server_key_version: "encryptionkey@0"
I20260621 07:04:23.848716 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:23.865129 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:23.866125 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:23.866472 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:23.887199 7242 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:43435
I20260621 07:04:23.887220 7307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:43435 every 8 connection(s)
I20260621 07:04:23.888881 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:23.889922 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7242
I20260621 07:04:23.890142 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:23.891084 1779 external_mini_cluster.cc:1428] Setting key 33d761f8c194e0ae6961fbd2436ea7a3
I20260621 07:04:23.894692 7308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:23.908576 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:23.893454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.1.188.254:43435)
0621 07:04:23.894754 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:23.894772 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:23.894839 (+ 67us) server_negotiation.cc:407] Connection header received
0621 07:04:23.898050 (+ 3211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:23.898078 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:23.898662 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:23.899079 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:23.899762 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.901220 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.903165 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:23.903993 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:23.904137 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:23.905571 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:23.905580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:23.905598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:04:23.906287 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:23.906379 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:04:23.906403 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:23.906507 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:23.906802 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 07:04:23.907352 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":400,"threads_started":1}
I20260621 07:04:23.909569 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca: Bootstrap starting.
I20260621 07:04:23.913166 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:04:23.913816 7308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:23.915041 7308 log.cc:824] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:23.918887 7308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca: No bootstrap required, opened a new log
I20260621 07:04:23.927263 7308 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } }
I20260621 07:04:23.927783 7308 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:23.927909 7308 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80fccbf1d81404abf80086a1f1c0cca, State: Initialized, Role: FOLLOWER
I20260621 07:04:23.928701 7308 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } }
I20260621 07:04:23.929055 7308 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:23.929215 7308 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:23.929419 7308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:23.931928 7308 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } }
I20260621 07:04:23.932560 7308 leader_election.cc:302] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80fccbf1d81404abf80086a1f1c0cca; no voters:
I20260621 07:04:23.933241 7308 leader_election.cc:288] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:23.933475 7313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:23.934846 7313 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [term 1 LEADER]: Becoming Leader. State: Replica: b80fccbf1d81404abf80086a1f1c0cca, State: Running, Role: LEADER
I20260621 07:04:23.935513 7313 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } }
I20260621 07:04:23.935899 7308 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:23.938643 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80fccbf1d81404abf80086a1f1c0cca. Latest consensus state: current_term: 1 leader_uuid: "b80fccbf1d81404abf80086a1f1c0cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } } }
I20260621 07:04:23.939002 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:23.938759 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80fccbf1d81404abf80086a1f1c0cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80fccbf1d81404abf80086a1f1c0cca" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43435 } } }
I20260621 07:04:23.939219 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:23.943554 7320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:23.949353 7320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:23.959302 7320 catalog_manager.cc:1347] Generated new cluster ID: f9e4c4298c2a4e55b741b9fb2d14c110
I20260621 07:04:23.959448 7320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:23.977205 7320 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:23.979221 7320 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:23.997054 7320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80fccbf1d81404abf80086a1f1c0cca: Generated new TSK 0
I20260621 07:04:23.998442 7320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:04:24.095743 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:24.017839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.1.188.254:43435)
0621 07:04:24.018108 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:24.018119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:24.018663 (+ 544us) server_negotiation.cc:407] Connection header received
0621 07:04:24.018963 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:24.018967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:24.019059 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:24.019207 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0621 07:04:24.019909 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:24.020715 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:24.022900 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:24.023479 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:24.023593 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:24.024242 (+ 649us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 07:04:24.094974 (+ 70732us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:04:24.095392 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260621 07:04:24.095960 7311 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 07:04:24.096699 7337 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:24.017786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:43435 (local address 127.0.0.1:51136)
0621 07:04:24.018513 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:24.018542 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:24.018729 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:24.019261 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:24.019267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:24.019295 (+ 28us) client_negotiation.cc:189] Negotiated authn=JWT
0621 07:04:24.019704 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:24.019719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:24.020871 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:24.020878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:24.022726 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:24.022741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:24.023669 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:24.023675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:24.023878 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:24.023945 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 07:04:24.095372 (+ 71427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 07:04:24.095791 (+ 419us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.188.254:43435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":376,"threads_started":1}
I20260621 07:04:24.098121 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7242
I20260621 07:04:24.099609 7303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:24.209779 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7242
2026-06-21T07:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (880 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (469 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (474 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:25 dist-test-slave-hlq1 krb5kdc[7352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:25 dist-test-slave-hlq1 krb5kdc[7352](info): set up 2 sockets
Jun 21 07:04:25 dist-test-slave-hlq1 krb5kdc[7352](info): commencing operation
krb5kdc: starting...
W20260621 07:04:26.264825 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:26 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:26.531749 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41889
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:26.909147 7365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:26.909658 7365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:26.909832 7365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:26.918062 7365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:26.918184 7365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:26.918236 7365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:26.918291 7365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:26.918339 7365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:26.929505 7365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:26.931252 7365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:26.932919 7365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:26.946265 7370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:26.946363 7373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:26.946229 7371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:26.947093 7365 server_base.cc:1034] running on GCE node
I20260621 07:04:26.948032 7365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:26.949694 7365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:26.950940 7365 hybrid_clock.cc:648] HybridClock initialized: now 1782025466950892 us; error 58 us; skew 500 ppm
I20260621 07:04:26.951344 7365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:26 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025466, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:26.954834 7365 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:26.956580 7365 webserver.cc:460] Webserver started at http://127.1.188.254:36161/ using document root <none> and password file <none>
I20260621 07:04:26.957430 7365 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:26.957588 7365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:26.957983 7365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:26.961668 7365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9de2814d866f4797a1d83b11d47ccf2d"
format_stamp: "Formatted at 2026-06-21 07:04:26 on dist-test-slave-hlq1"
server_key: "3a2b66663f7bac892ef5b27eb67354cc"
server_key_iv: "ae9454f15ec9e57715cd48472f118bbf"
server_key_version: "encryptionkey@0"
I20260621 07:04:26.962572 7365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9de2814d866f4797a1d83b11d47ccf2d"
format_stamp: "Formatted at 2026-06-21 07:04:26 on dist-test-slave-hlq1"
server_key: "3a2b66663f7bac892ef5b27eb67354cc"
server_key_iv: "ae9454f15ec9e57715cd48472f118bbf"
server_key_version: "encryptionkey@0"
I20260621 07:04:26.968869 7365 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 07:04:26.972878 7380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:26.974499 7365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:26.974697 7365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9de2814d866f4797a1d83b11d47ccf2d"
format_stamp: "Formatted at 2026-06-21 07:04:26 on dist-test-slave-hlq1"
server_key: "3a2b66663f7bac892ef5b27eb67354cc"
server_key_iv: "ae9454f15ec9e57715cd48472f118bbf"
server_key_version: "encryptionkey@0"
I20260621 07:04:26.974925 7365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:26.988646 7365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:26.993417 7365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:26.993741 7365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:27.014810 7365 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41889
I20260621 07:04:27.014829 7431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41889 every 8 connection(s)
I20260621 07:04:27.016606 7365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:27.021404 7432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:27.026502 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7365
I20260621 07:04:27.026780 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:27.027880 1779 external_mini_cluster.cc:1428] Setting key 10014c4c155186a304df98549c597ee6
I20260621 07:04:27.037657 7432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d: Bootstrap starting.
I20260621 07:04:27.042408 7432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:27.044082 7432 log.cc:824] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:04:27 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:27.047602 7432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d: No bootstrap required, opened a new log
I20260621 07:04:27.058321 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:27.030801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.1.188.254:41889)
0621 07:04:27.031994 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:27.032013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:27.032589 (+ 576us) server_negotiation.cc:407] Connection header received
0621 07:04:27.035538 (+ 2949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:27.035579 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:27.036310 (+ 731us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:27.036922 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:27.038649 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.040933 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.042359 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.043560 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.043772 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:27.048504 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:27.048515 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:27.048538 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:27.048590 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:27.052995 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:27.053734 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:27.053743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:27.053754 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:27.053988 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:27.054363 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:27.054371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:27.054379 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:27.055236 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:27.055599 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:27.056000 (+ 401us) server_negotiation.cc:299] Negotiation successful
0621 07:04:27.056533 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":506,"threads_started":1}
I20260621 07:04:27.059300 7432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } }
I20260621 07:04:27.059831 7432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:27.059967 7432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de2814d866f4797a1d83b11d47ccf2d, State: Initialized, Role: FOLLOWER
I20260621 07:04:27.060680 7432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } }
I20260621 07:04:27.060943 7432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:27.061120 7432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:27.061313 7432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:27.063776 7432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } }
I20260621 07:04:27.064323 7432 leader_election.cc:302] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de2814d866f4797a1d83b11d47ccf2d; no voters:
I20260621 07:04:27.065084 7432 leader_election.cc:288] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:27.065196 7437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:27.066007 7437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [term 1 LEADER]: Becoming Leader. State: Replica: 9de2814d866f4797a1d83b11d47ccf2d, State: Running, Role: LEADER
I20260621 07:04:27.066591 7437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } }
I20260621 07:04:27.067673 7432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:27.070039 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de2814d866f4797a1d83b11d47ccf2d. Latest consensus state: current_term: 1 leader_uuid: "9de2814d866f4797a1d83b11d47ccf2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } } }
I20260621 07:04:27.070067 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de2814d866f4797a1d83b11d47ccf2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de2814d866f4797a1d83b11d47ccf2d" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41889 } } }
I20260621 07:04:27.070482 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:27.070482 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:27.072180 7444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:27.078116 7444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:27.089273 7444 catalog_manager.cc:1347] Generated new cluster ID: 51ea2ae787bb4cf8af51c04284340168
I20260621 07:04:27.089437 7444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:27.117236 7444 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:27.119668 7444 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:27.135000 7444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de2814d866f4797a1d83b11d47ccf2d: Generated new TSK 0
I20260621 07:04:27.136106 7444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:27.202630 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:27.567742 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:27.568085 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:27.568226 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:27.575975 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:27.576090 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:27.576319 7458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:27.586875 7458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:27.588332 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:27.589684 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:27.603358 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:27.603433 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:27.603338 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:27.604020 7458 server_base.cc:1034] running on GCE node
I20260621 07:04:27.604648 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:27.605624 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:27.606885 7458 hybrid_clock.cc:648] HybridClock initialized: now 1782025467606834 us; error 65 us; skew 500 ppm
I20260621 07:04:27.607223 7458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:27 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025467, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:27.610533 7458 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:27.612052 7458 webserver.cc:460] Webserver started at http://127.1.188.193:44271/ using document root <none> and password file <none>
I20260621 07:04:27.612787 7458 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:27.612932 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:27.613250 7458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:27.616220 7458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "c36298768426406e93012ae2b7ce959f"
format_stamp: "Formatted at 2026-06-21 07:04:27 on dist-test-slave-hlq1"
server_key: "29a5afe8c3a72a89509fd6ed8bcf5270"
server_key_iv: "2618b436d9a2272d7570c386127bed15"
server_key_version: "encryptionkey@0"
I20260621 07:04:27.617005 7458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "c36298768426406e93012ae2b7ce959f"
format_stamp: "Formatted at 2026-06-21 07:04:27 on dist-test-slave-hlq1"
server_key: "29a5afe8c3a72a89509fd6ed8bcf5270"
server_key_iv: "2618b436d9a2272d7570c386127bed15"
server_key_version: "encryptionkey@0"
I20260621 07:04:27.623275 7458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260621 07:04:27.627775 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:27.629045 7458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:04:27.629223 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "c36298768426406e93012ae2b7ce959f"
format_stamp: "Formatted at 2026-06-21 07:04:27 on dist-test-slave-hlq1"
server_key: "29a5afe8c3a72a89509fd6ed8bcf5270"
server_key_iv: "2618b436d9a2272d7570c386127bed15"
server_key_version: "encryptionkey@0"
I20260621 07:04:27.629428 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:27.654577 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:27.658933 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:27.659235 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:27.660288 7458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:27.661680 7458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:27.661783 7458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:27.661926 7458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:27.662000 7458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:27.696960 7458 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:36843
I20260621 07:04:27.697018 7585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:36843 every 8 connection(s)
I20260621 07:04:27.698290 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:27.707370 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7458
I20260621 07:04:27.707647 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:27.708650 1779 external_mini_cluster.cc:1428] Setting key 038f85c2e98d00a37ab5fcc7a1e5785a
Jun 21 07:04:27 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025467, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:27.720878 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:27.700984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:44585 (local address 127.1.188.254:41889)
0621 07:04:27.701510 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:27.701517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:27.702387 (+ 870us) server_negotiation.cc:407] Connection header received
0621 07:04:27.704204 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:27.704209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:27.704289 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:27.704392 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:27.706015 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.706870 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.709526 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.710087 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.710220 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:27.714581 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:27.714587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:27.714608 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:27.714640 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:27.717350 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:27.718085 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:27.718093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:27.718109 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:27.718272 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:27.718779 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:27.718786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:27.718793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:27.719186 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:27.719338 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:27.720121 (+ 783us) server_negotiation.cc:299] Negotiation successful
0621 07:04:27.720380 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":270,"threads_started":1}
I20260621 07:04:27.721525 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:27.701464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41889 (local address 127.1.188.193:44585)
0621 07:04:27.702221 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:27.702264 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:27.703934 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:27.704668 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:27.704676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:27.705235 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:27.705832 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:27.705851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.707058 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.707065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:27.709379 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:27.709391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:27.710296 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:27.710301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:27.710495 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:27.711306 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:27.711336 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:27.714345 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:27.717499 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:27.717507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:27.717519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:27.717950 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:27.718449 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:27.718453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:27.718456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:27.718646 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:27.719317 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:27.719323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:27.719724 (+ 401us) client_negotiation.cc:769] Sending connection context
0621 07:04:27.719957 (+ 233us) client_negotiation.cc:240] Negotiation successful
0621 07:04:27.720389 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":304,"threads_started":1}
I20260621 07:04:27.723227 7586 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41889
I20260621 07:04:27.723588 7586 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:27.724335 7586 heartbeater.cc:511] Master 127.1.188.254:41889 requested a full tablet report, sending...
I20260621 07:04:27.726491 7397 ts_manager.cc:194] Registered new tserver with Master: c36298768426406e93012ae2b7ce959f (127.1.188.193:36843)
I20260621 07:04:27.728087 7397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:44585
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:27.757081 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:28.086637 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:28.086948 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:28.087117 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:28.094475 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:28.094633 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:28.094806 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:28.104671 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:28.106033 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:28.107340 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:28.120617 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:28.120790 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:28.121129 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:28.121539 7592 server_base.cc:1034] running on GCE node
I20260621 07:04:28.122172 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:28.123225 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:28.124457 7592 hybrid_clock.cc:648] HybridClock initialized: now 1782025468124393 us; error 59 us; skew 500 ppm
I20260621 07:04:28.124789 7592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.128209 7592 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:28.129751 7592 webserver.cc:460] Webserver started at http://127.1.188.194:44719/ using document root <none> and password file <none>
I20260621 07:04:28.130467 7592 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:28.130617 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:28.130937 7592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:28.134205 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "3c77d2132ee7434f9bd60a317c4c9f8e"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "f8c45bc0ec2da10abd4138dde70218c5"
server_key_iv: "63e36c2ccd3ff98a2e7e6e7cda57ff5d"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.135049 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "3c77d2132ee7434f9bd60a317c4c9f8e"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "f8c45bc0ec2da10abd4138dde70218c5"
server_key_iv: "63e36c2ccd3ff98a2e7e6e7cda57ff5d"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.140362 7592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:04:28.144559 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.146077 7592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 07:04:28.146242 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "3c77d2132ee7434f9bd60a317c4c9f8e"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "f8c45bc0ec2da10abd4138dde70218c5"
server_key_iv: "63e36c2ccd3ff98a2e7e6e7cda57ff5d"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.146456 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:28.159026 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:28.163335 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:28.163662 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:28.164595 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:28.165920 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:28.165997 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.166090 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:28.166141 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.203034 7592 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:44333
I20260621 07:04:28.203146 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:44333 every 8 connection(s)
I20260621 07:04:28.204452 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:28.207748 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7592
I20260621 07:04:28.207926 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:28.208397 1779 external_mini_cluster.cc:1428] Setting key d2ee71eac6078b20976b12f7cd2832ef
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:28.228755 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.207059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:44697 (local address 127.1.188.254:41889)
0621 07:04:28.207287 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.207293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.208718 (+ 1425us) server_negotiation.cc:407] Connection header received
0621 07:04:28.210749 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.210755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.210838 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.210947 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.212528 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.213659 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.215261 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.216049 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.216204 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.220543 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.220554 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.220562 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.220613 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.224756 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.225633 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.225642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.225649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.225840 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.226361 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.226370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.226376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.226837 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.227024 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.227917 (+ 893us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.228230 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 07:04:28.229300 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.207684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41889 (local address 127.1.188.194:44697)
0621 07:04:28.208516 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:28.208566 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:28.210431 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:28.211162 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:28.211171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:28.211774 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:28.212350 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:28.212367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.213836 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.213840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:28.215101 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:28.215113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.216284 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.216289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:28.216552 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.217361 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:28.217386 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:28.220287 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:28.224945 (+ 4658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:28.224952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:28.224967 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:28.225449 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:28.225992 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:28.225996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:28.225998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:28.226219 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:28.227015 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:28.227022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:28.227458 (+ 436us) client_negotiation.cc:769] Sending connection context
0621 07:04:28.227724 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 07:04:28.228139 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":373,"threads_started":1}
I20260621 07:04:28.230906 7720 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41889
I20260621 07:04:28.231276 7720 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:04:28.232121 7720 heartbeater.cc:511] Master 127.1.188.254:41889 requested a full tablet report, sending...
I20260621 07:04:28.233716 7397 ts_manager.cc:194] Registered new tserver with Master: 3c77d2132ee7434f9bd60a317c4c9f8e (127.1.188.194:44333)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:04:28.234819 7397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:44697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:28.259075 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:28.595602 7725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:28.595897 7725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:28.596061 7725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:28.603984 7725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:28.604099 7725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:28.604270 7725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:28.614315 7725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:28.615708 7725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:28.616995 7725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:28.629822 7730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:28.629858 7733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:28.630007 7731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:28.630950 7725 server_base.cc:1034] running on GCE node
I20260621 07:04:28.631665 7725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:28.632627 7725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:28.633884 7725 hybrid_clock.cc:648] HybridClock initialized: now 1782025468633848 us; error 55 us; skew 500 ppm
I20260621 07:04:28.634297 7725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.637485 7725 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:28.639016 7725 webserver.cc:460] Webserver started at http://127.1.188.195:37283/ using document root <none> and password file <none>
I20260621 07:04:28.639775 7725 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:28.639883 7725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:28.640172 7725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:28.643026 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "30ff94876969450e99581776e3fc7db6"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "3dff16f4e0a81e32662b6e6b438ad64b"
server_key_iv: "442148034bf2e4414c5ed7814ba1f897"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.643829 7725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "30ff94876969450e99581776e3fc7db6"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "3dff16f4e0a81e32662b6e6b438ad64b"
server_key_iv: "442148034bf2e4414c5ed7814ba1f897"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.648875 7725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:04:28.652542 7740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.653908 7725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:04:28.654091 7725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "30ff94876969450e99581776e3fc7db6"
format_stamp: "Formatted at 2026-06-21 07:04:28 on dist-test-slave-hlq1"
server_key: "3dff16f4e0a81e32662b6e6b438ad64b"
server_key_iv: "442148034bf2e4414c5ed7814ba1f897"
server_key_version: "encryptionkey@0"
I20260621 07:04:28.654297 7725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:28.670490 7725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:28.674865 7725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:28.675181 7725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:28.676234 7725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:28.677796 7725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:28.677901 7725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.678043 7725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:28.678122 7725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:28.713267 7725 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:41651
I20260621 07:04:28.713302 7852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:41651 every 8 connection(s)
I20260621 07:04:28.714563 7725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:28.719581 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7725
I20260621 07:04:28.719771 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:28.720279 1779 external_mini_cluster.cc:1428] Setting key 17d53cdeca8234184c01444169a0fc61
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:28.731127 7586 heartbeater.cc:503] Master 127.1.188.254:41889 was elected leader, sending a full tablet report...
I20260621 07:04:28.737253 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.717647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:36463 (local address 127.1.188.254:41889)
0621 07:04:28.717857 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.717863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.719053 (+ 1190us) server_negotiation.cc:407] Connection header received
0621 07:04:28.721943 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.721952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.722049 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.722129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.724023 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.724869 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.726399 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.726828 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.726941 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.730512 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.730517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.730521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.730554 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.733690 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.734547 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.734551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.734554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.734715 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.735457 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.735463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.735467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.735862 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.735995 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.736835 (+ 840us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.737006 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:04:28.738308 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.718134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41889 (local address 127.1.188.195:36463)
0621 07:04:28.718893 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:28.718931 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:28.721553 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:28.722629 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:28.722637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:28.723151 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:28.723822 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:28.723847 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.725008 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.725012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:28.726266 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:28.726277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.727037 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.727042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:28.727283 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.728005 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:28.728029 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:28.730301 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:28.733818 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:28.733824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:28.733884 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:28.734392 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:28.734898 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:28.734904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:28.734908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:28.735249 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:28.736013 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:28.736022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:28.736442 (+ 420us) client_negotiation.cc:769] Sending connection context
0621 07:04:28.736696 (+ 254us) client_negotiation.cc:240] Negotiation successful
0621 07:04:28.737113 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
I20260621 07:04:28.739706 7853 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41889
I20260621 07:04:28.739988 7853 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:28.740703 7853 heartbeater.cc:511] Master 127.1.188.254:41889 requested a full tablet report, sending...
I20260621 07:04:28.742054 7397 ts_manager.cc:194] Registered new tserver with Master: 30ff94876969450e99581776e3fc7db6 (127.1.188.195:41651)
I20260621 07:04:28.742867 7397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:36463
I20260621 07:04:28.748915 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/test-admin.keytab.
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.777077 1779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260621 07:04:28.782605 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:28.780083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.1.188.193:36843)
0621 07:04:28.780837 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.780844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.780870 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:28.780929 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.780936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.781152 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.781291 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.782230 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36018: BlockingRecv error: recv got EOF from 127.0.0.1:36018 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":340,"threads_started":1}
W20260621 07:04:28.782998 1779 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.784614 1779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.786026 1779 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:28.803107 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.788126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.1.188.193:36843)
0621 07:04:28.788309 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.788313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.788510 (+ 197us) server_negotiation.cc:407] Connection header received
0621 07:04:28.788631 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.788635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.788701 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.788816 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.789857 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.790959 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.792395 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.792978 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.793085 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.797032 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.797044 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.797060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.797148 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.800517 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.801233 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.801240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.801243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.801365 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.801698 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.801701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.801703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.802061 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.802282 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.802619 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.802828 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 07:04:28.809158 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:28.807250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.1.188.193:36843)
0621 07:04:28.807484 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.807497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.807806 (+ 309us) server_negotiation.cc:407] Connection header received
0621 07:04:28.807985 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.807990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.808056 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.808167 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.808874 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36034: BlockingRecv error: recv got EOF from 127.0.0.1:36034 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 07:04:28.809689 1779 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc)
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.811169 1779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.812496 1779 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:28.826962 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.813921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.1.188.193:36843)
0621 07:04:28.814211 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.814216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.814466 (+ 250us) server_negotiation.cc:407] Connection header received
0621 07:04:28.814673 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.814678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.814744 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.814857 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.815770 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.816484 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.817707 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.818112 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.818261 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.821958 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.821963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.821966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.822015 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.824483 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.825128 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.825133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.825139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.825264 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.825641 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.825646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.825650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.825918 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.826189 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.826465 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.826683 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260621 07:04:28.831540 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:28.829630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.1.188.193:36843)
0621 07:04:28.829862 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.829871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.830182 (+ 311us) server_negotiation.cc:407] Connection header received
0621 07:04:28.830313 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.830316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.830394 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.830478 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.831260 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36052: BlockingRecv error: recv got EOF from 127.0.0.1:36052 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 07:04:28.831655 1779 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782025418728693-1779-0/krb5kdc/krb5cc' of type FILE
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.833246 1779 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:28.834698 1779 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:28.856992 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.836093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.1.188.193:36843)
0621 07:04:28.836337 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.836345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.836672 (+ 327us) server_negotiation.cc:407] Connection header received
0621 07:04:28.836861 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.836871 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.836927 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.837037 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.837991 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.838701 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.840025 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.840450 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.840592 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.851666 (+ 11074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.851672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.851676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.851715 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.854415 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.855134 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.855140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.855143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.855305 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.855739 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.855743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.855746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.856020 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.856269 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.856550 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.856733 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:04:28.858850 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7458
I20260621 07:04:28.870895 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.859574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.1.188.193:36843)
0621 07:04:28.859737 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.859741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.860172 (+ 431us) server_negotiation.cc:407] Connection header received
0621 07:04:28.860269 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.860273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.860332 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.860417 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.861392 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.862052 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.863335 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.863838 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.863931 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.865946 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.865950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.865953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.865978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.868323 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.869142 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.869146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.869149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.869323 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.869702 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.869705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.869707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.869957 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.870193 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.870468 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.870626 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:04:28.871562 7581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:28.960662 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7458
I20260621 07:04:28.976810 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7592
Jun 21 07:04:28 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:28.993083 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:28.977700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.1.188.194:44333)
0621 07:04:28.978246 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:28.978254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:28.978278 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:04:28.978337 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:28.978343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:28.978572 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:28.978751 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:28.979850 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.980821 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.982063 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:28.982594 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:28.982740 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:28.986742 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:28.986754 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:28.986770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:28.986820 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:28.989884 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.990704 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.990711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.990714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.990870 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:28.991339 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:28.991343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:28.991346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:28.991877 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:28.992199 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:28.992517 (+ 318us) server_negotiation.cc:299] Negotiation successful
0621 07:04:28.992751 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":297,"threads_started":1}
I20260621 07:04:28.994436 7715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:29.079254 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7592
I20260621 07:04:29.095211 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7725
Jun 21 07:04:29 dist-test-slave-hlq1 krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:29.111356 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:29.096235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.1.188.195:41651)
0621 07:04:29.097018 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:29.097025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:29.097056 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:04:29.097154 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:29.097160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:29.097383 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:29.097513 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:29.098518 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:29.099597 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:29.100961 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:29.101545 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:29.101644 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:29.105751 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:29.105762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:29.105775 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:29.105812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:29.108667 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:29.109333 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:29.109340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:29.109342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:29.109457 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:29.109876 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:29.109880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:29.109882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:29.110242 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:29.110483 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:29.110710 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 07:04:29.110991 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":490,"threads_started":1}
I20260621 07:04:29.112673 7848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:29.196106 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7725
I20260621 07:04:29.212091 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7365
I20260621 07:04:29.213313 7427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:29.316946 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7365
2026-06-21T07:04:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4165 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:29 dist-test-slave-hlq1 krb5kdc[7880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:29 dist-test-slave-hlq1 krb5kdc[7880](info): set up 2 sockets
Jun 21 07:04:29 dist-test-slave-hlq1 krb5kdc[7880](info): commencing operation
krb5kdc: starting...
W20260621 07:04:30.436568 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:30 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab.
Entry for principal oryx/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab.
I20260621 07:04:30.671589 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37109
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab
--principal=oryx/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:31.044654 7893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:31.045058 7893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:31.045261 7893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:31.053298 7893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:31.053449 7893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:31.053562 7893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:31.053683 7893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:31.053776 7893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:31.064898 7893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab
--principal=oryx/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:31.066669 7893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:31.068502 7893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:31.082423 7899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:31.082450 7898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:31.083190 7901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:31.083560 7893 server_base.cc:1034] running on GCE node
I20260621 07:04:31.084419 7893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:31.086108 7893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:31.087343 7893 hybrid_clock.cc:648] HybridClock initialized: now 1782025471087304 us; error 57 us; skew 500 ppm
I20260621 07:04:31.087740 7893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:31 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025471, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:31.091348 7893 init.cc:377] Logged in from keytab as oryx/127.1.188.254@KRBTEST.COM (short username oryx)
I20260621 07:04:31.093053 7893 webserver.cc:460] Webserver started at http://127.1.188.254:43407/ using document root <none> and password file <none>
I20260621 07:04:31.093868 7893 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:31.094007 7893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:31.094372 7893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:31.097870 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "6245152a3fc4468fbbc9bd6194d309aa"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "21f36ebf8a9b47667c92cf3582b9908a"
server_key_iv: "be4a0bcbb7ad2f92bcfc47b01430fbec"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.098718 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "6245152a3fc4468fbbc9bd6194d309aa"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "21f36ebf8a9b47667c92cf3582b9908a"
server_key_iv: "be4a0bcbb7ad2f92bcfc47b01430fbec"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.104678 7893 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260621 07:04:31.108609 7908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:31.110069 7893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:04:31.110255 7893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "6245152a3fc4468fbbc9bd6194d309aa"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "21f36ebf8a9b47667c92cf3582b9908a"
server_key_iv: "be4a0bcbb7ad2f92bcfc47b01430fbec"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.110473 7893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:31.134814 7893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:31.140053 7893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:31.140413 7893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:31.161417 7893 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:37109
I20260621 07:04:31.161453 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:37109 every 8 connection(s)
I20260621 07:04:31.163030 7893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:31.165020 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7893
I20260621 07:04:31.165241 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:31.165964 1779 external_mini_cluster.cc:1428] Setting key 0bd94495a0b16d4c56b8e51fa893baa0
I20260621 07:04:31.168062 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 07:04:31 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:31.183465 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Bootstrap starting.
I20260621 07:04:31.187357 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:31.189023 7960 log.cc:824] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:31.192168 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:31.168308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.1.188.254:37109)
0621 07:04:31.169177 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:31.169190 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:31.169239 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:04:31.170986 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:31.171012 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:31.171539 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:31.171935 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:31.173994 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.175538 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.177321 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.178026 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.178170 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:31.183299 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:31.183309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:31.183328 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:31.183384 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:31.186795 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:31.187687 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:31.187699 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:31.187709 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:31.188008 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:31.188412 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:31.188417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:31.188421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:31.189295 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:31.189622 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:31.190083 (+ 461us) server_negotiation.cc:299] Negotiation successful
0621 07:04:31.190570 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":392,"threads_started":1}
I20260621 07:04:31.193176 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: No bootstrap required, opened a new log
I20260621 07:04:31.201534 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } }
I20260621 07:04:31.201925 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:31.202003 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6245152a3fc4468fbbc9bd6194d309aa, State: Initialized, Role: FOLLOWER
I20260621 07:04:31.202657 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } }
I20260621 07:04:31.202884 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:31.202991 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:31.203126 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:31.205437 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } }
I20260621 07:04:31.206030 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6245152a3fc4468fbbc9bd6194d309aa; no voters:
I20260621 07:04:31.206856 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:31.206988 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:31.207700 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 LEADER]: Becoming Leader. State: Replica: 6245152a3fc4468fbbc9bd6194d309aa, State: Running, Role: LEADER
I20260621 07:04:31.208315 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } }
I20260621 07:04:31.209435 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:31.211525 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } }
I20260621 07:04:31.211884 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:31.212864 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6245152a3fc4468fbbc9bd6194d309aa. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } }
I20260621 07:04:31.213268 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:31.214434 7970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:31.221455 7970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:31.232858 7970 catalog_manager.cc:1347] Generated new cluster ID: 091540ad3b1442a7bc021c36f7783afe
I20260621 07:04:31.233036 7970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:31.254444 7970 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:31.256737 7970 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:31.267488 7970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Generated new TSK 0
I20260621 07:04:31.268842 7970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
I20260621 07:04:31.330674 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:31.690043 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:31.690347 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:31.690510 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:31.698493 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:31.698624 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:31.698746 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:31.708983 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:31.710474 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:31.711938 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:31.725723 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:31.725643 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:31.725726 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:31.726619 7986 server_base.cc:1034] running on GCE node
I20260621 07:04:31.727358 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:31.728379 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:31.729586 7986 hybrid_clock.cc:648] HybridClock initialized: now 1782025471729519 us; error 55 us; skew 500 ppm
I20260621 07:04:31.729872 7986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:31 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025471, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:31.733338 7986 init.cc:377] Logged in from keytab as oryx/127.1.188.193@KRBTEST.COM (short username oryx)
I20260621 07:04:31.734835 7986 webserver.cc:460] Webserver started at http://127.1.188.193:41625/ using document root <none> and password file <none>
I20260621 07:04:31.735642 7986 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:31.735772 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:31.736106 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:31.739044 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "e68f9669d85a4a36bbbc17470fbe03fc"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "3b3012c01be447dea925715fd025af60"
server_key_iv: "2dc7a7a5e2ad400d8a259fbebb15701a"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.739888 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "e68f9669d85a4a36bbbc17470fbe03fc"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "3b3012c01be447dea925715fd025af60"
server_key_iv: "2dc7a7a5e2ad400d8a259fbebb15701a"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.745491 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:04:31.749351 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:31.750775 7986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:04:31.750959 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "e68f9669d85a4a36bbbc17470fbe03fc"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "3b3012c01be447dea925715fd025af60"
server_key_iv: "2dc7a7a5e2ad400d8a259fbebb15701a"
server_key_version: "encryptionkey@0"
I20260621 07:04:31.751211 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:31.763005 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:31.767050 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:31.767357 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:31.768299 7986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:31.769613 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:31.769711 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:31.769857 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:31.769934 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:31.803697 7986 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:38565
I20260621 07:04:31.803752 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:38565 every 8 connection(s)
I20260621 07:04:31.805040 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:31.814697 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 7986
I20260621 07:04:31.814924 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:31.815810 1779 external_mini_cluster.cc:1428] Setting key 111a38ea31ce6df4830f5b75fa0f854a
Jun 21 07:04:31 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025471, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:31.825138 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:31.807599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:53407 (local address 127.1.188.254:37109)
0621 07:04:31.808039 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:31.808045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:31.808944 (+ 899us) server_negotiation.cc:407] Connection header received
0621 07:04:31.810772 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:31.810796 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:31.810888 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:31.811007 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:31.812621 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.813393 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.815157 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.815620 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.815802 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:31.819467 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:31.819473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:31.819476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:31.819508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:31.822056 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:31.822821 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:31.822827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:31.822830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:31.822951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:31.823374 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:31.823382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:31.823386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:31.823679 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:31.823844 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:31.824530 (+ 686us) server_negotiation.cc:299] Negotiation successful
0621 07:04:31.824750 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260621 07:04:31.826045 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:31.808154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:53407)
0621 07:04:31.808791 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:31.808826 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:31.810519 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:31.811285 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:31.811294 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:31.811892 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:31.812436 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:31.812452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.813546 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.813550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:31.815000 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:31.815032 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:31.815845 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:31.815852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:31.816154 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:31.816837 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:31.816858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:31.819191 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:31.822212 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:31.822222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:31.822239 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:31.822698 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:31.823066 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:31.823071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:31.823075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:31.823256 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:31.823810 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:31.823819 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:31.824164 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 07:04:31.824437 (+ 273us) client_negotiation.cc:240] Negotiation successful
0621 07:04:31.824955 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":241,"threads_started":1}
I20260621 07:04:31.827761 8114 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:31.828102 8114 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:31.828794 8114 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:31.831007 7925 ts_manager.cc:194] Registered new tserver with Master: e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:31.832580 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:53407
WARNING: no policy specified for oryx/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
I20260621 07:04:31.860850 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:32.187912 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:32.188197 8120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:32.188289 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:32.195636 8120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:32.195742 8120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:32.195853 8120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:32.205408 8120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:32.206642 8120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:32.207959 8120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:32.221691 8128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:32.221774 8126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:32.221958 8125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:32.222847 8120 server_base.cc:1034] running on GCE node
I20260621 07:04:32.223489 8120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:32.224436 8120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:32.225673 8120 hybrid_clock.cc:648] HybridClock initialized: now 1782025472225633 us; error 37 us; skew 500 ppm
I20260621 07:04:32.225993 8120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:32 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025472, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:32.229312 8120 init.cc:377] Logged in from keytab as oryx/127.1.188.194@KRBTEST.COM (short username oryx)
I20260621 07:04:32.230850 8120 webserver.cc:460] Webserver started at http://127.1.188.194:40247/ using document root <none> and password file <none>
I20260621 07:04:32.231643 8120 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:32.231783 8120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:32.232087 8120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:32.234964 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "ecf10dfd450b4ed48fe793db0ed11988"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "5d2683d33a502c63548624e145000036"
server_key_iv: "70a36eea89578474c240fc58b71fdc1b"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.235791 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "ecf10dfd450b4ed48fe793db0ed11988"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "5d2683d33a502c63548624e145000036"
server_key_iv: "70a36eea89578474c240fc58b71fdc1b"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.240829 8120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:04:32.244177 8135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:32.245496 8120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:32.245684 8120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "ecf10dfd450b4ed48fe793db0ed11988"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "5d2683d33a502c63548624e145000036"
server_key_iv: "70a36eea89578474c240fc58b71fdc1b"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.245885 8120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:32.274968 8120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:32.279518 8120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:32.279812 8120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:32.280884 8120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:32.282260 8120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:32.282363 8120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:32.282509 8120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:32.282581 8120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:32.320144 8120 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43957
I20260621 07:04:32.320256 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43957 every 8 connection(s)
I20260621 07:04:32.321466 8120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:32.321838 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8120
I20260621 07:04:32.322022 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:32.322629 1779 external_mini_cluster.cc:1428] Setting key 770ca9f9107a06497eac0ecb6f2a2a1c
Jun 21 07:04:32 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025472, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:04:32.345304 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:32.324166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:47669 (local address 127.1.188.254:37109)
0621 07:04:32.324331 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:32.324335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:32.325838 (+ 1503us) server_negotiation.cc:407] Connection header received
0621 07:04:32.328741 (+ 2903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:32.328745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:32.328812 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:32.328898 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:32.330982 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.331800 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.333425 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.334219 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.334364 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:32.337934 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:32.337943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:32.337951 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:32.337998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:32.341820 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:32.342548 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:32.342556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:32.342563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:32.342730 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:32.343148 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:32.343156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:32.343168 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:32.343576 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:32.343738 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:32.344626 (+ 888us) server_negotiation.cc:299] Negotiation successful
0621 07:04:32.344888 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:04:32.346299 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:32.324777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:47669)
0621 07:04:32.325647 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:32.325707 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:32.328433 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:32.329173 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:32.329187 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:32.329975 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:32.330794 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:32.330820 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.331967 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.331975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:32.333274 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:32.333287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.334450 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.334455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:32.334660 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:32.335390 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:32.335486 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:32.337697 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:32.341979 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:32.341986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:32.342000 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:32.342420 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:32.342852 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:32.342855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:32.342857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:32.343032 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:32.343714 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:32.343721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:32.344043 (+ 322us) client_negotiation.cc:769] Sending connection context
0621 07:04:32.344481 (+ 438us) client_negotiation.cc:240] Negotiation successful
0621 07:04:32.344904 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.195@KRBTEST.COM" created.
I20260621 07:04:32.347910 8248 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:32.348233 8248 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:32.348935 8248 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:32.350656 7925 ts_manager.cc:194] Registered new tserver with Master: ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957)
I20260621 07:04:32.351598 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
I20260621 07:04:32.370656 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:32.747161 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:32.747642 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:32.747841 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:32.756362 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:32.756539 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:32.756752 8253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:32.767369 8253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:32.769055 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:32.770705 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:32.785526 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:32.785637 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:32.785534 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:32.786696 8253 server_base.cc:1034] running on GCE node
I20260621 07:04:32.787441 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:32.788463 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:32.789666 8253 hybrid_clock.cc:648] HybridClock initialized: now 1782025472789616 us; error 44 us; skew 500 ppm
I20260621 07:04:32.790020 8253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:32 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025472, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:32.793532 8253 init.cc:377] Logged in from keytab as oryx/127.1.188.195@KRBTEST.COM (short username oryx)
I20260621 07:04:32.795313 8253 webserver.cc:460] Webserver started at http://127.1.188.195:37289/ using document root <none> and password file <none>
I20260621 07:04:32.796126 8253 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:32.796231 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:32.796501 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:32.799762 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "5d753a88e97545f1958c842e6c28012a"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "68bcff1284933cfbb70a0d8886cf9de8"
server_key_iv: "17ea62f80cb042272bc1cfa30e0b4d66"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.800532 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "5d753a88e97545f1958c842e6c28012a"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "68bcff1284933cfbb70a0d8886cf9de8"
server_key_iv: "17ea62f80cb042272bc1cfa30e0b4d66"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.806581 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260621 07:04:32.810591 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:32.812219 8253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:32.812394 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "5d753a88e97545f1958c842e6c28012a"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "68bcff1284933cfbb70a0d8886cf9de8"
server_key_iv: "17ea62f80cb042272bc1cfa30e0b4d66"
server_key_version: "encryptionkey@0"
I20260621 07:04:32.812602 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:32.830571 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:32.835754 8114 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
I20260621 07:04:32.836020 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:32.836349 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:32.837638 8253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:32.839452 8253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:32.839569 8253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 07:04:32.839733 8253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:32.839808 8253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:32.876147 8253 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:37761
I20260621 07:04:32.876180 8380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:37761 every 8 connection(s)
I20260621 07:04:32.877681 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:32.885736 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8253
I20260621 07:04:32.885937 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:32.886601 1779 external_mini_cluster.cc:1428] Setting key 4296d538aeb916d19d2027a2ace5b7c2
Jun 21 07:04:32 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025472, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:32.901187 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:32.880651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:34777 (local address 127.1.188.254:37109)
0621 07:04:32.881186 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:32.881193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:32.882380 (+ 1187us) server_negotiation.cc:407] Connection header received
0621 07:04:32.884411 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:32.884415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:32.884499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:32.884607 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:32.886658 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.887722 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.890244 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.890673 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.890779 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:32.894782 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:32.894793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:32.894799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:32.894850 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:32.897731 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:32.898548 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:32.898553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:32.898555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:32.898713 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:32.899197 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:32.899201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:32.899203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:32.899633 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:32.899776 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:32.900559 (+ 783us) server_negotiation.cc:299] Negotiation successful
0621 07:04:32.900782 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260621 07:04:32.902707 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:32.881408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:34777)
0621 07:04:32.882209 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:32.882251 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:32.884135 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:32.884830 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:32.884839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:32.885559 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:32.886449 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:32.886477 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.888003 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.888010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:32.890070 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:32.890088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:32.890909 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:32.890915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:32.891172 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:32.892035 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:32.892061 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:32.894487 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:32.897885 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:32.897893 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:32.897917 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:32.898406 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:32.898846 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:32.898859 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:32.898862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:32.899076 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:32.899772 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:32.899780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:32.900173 (+ 393us) client_negotiation.cc:769] Sending connection context
0621 07:04:32.900463 (+ 290us) client_negotiation.cc:240] Negotiation successful
0621 07:04:32.901110 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":299,"threads_started":1}
I20260621 07:04:32.904388 8381 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:32.904775 8381 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:32.905644 8381 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:32.907194 7925 ts_manager.cc:194] Registered new tserver with Master: 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
I20260621 07:04:32.908171 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:34777
I20260621 07:04:32.915889 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.188.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.253.keytab.
Entry for principal oryx/127.1.188.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.253.keytab.
I20260621 07:04:32.977146 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:43365
--webserver_interface=127.1.188.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.253.keytab
--principal=oryx/127.1.188.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37109,127.1.188.253:43365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:33.354745 8248 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
W20260621 07:04:33.401196 8387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:33.401625 8387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:33.401798 8387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:33.410030 8387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:33.410144 8387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:33.410190 8387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:33.410248 8387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:33.410290 8387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:33.420815 8387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37109,127.1.188.253:43365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.253.keytab
--principal=oryx/127.1.188.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.188.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
W20260621 07:04:33.421166 8387 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.188.254:37109,127.1.188.253:43365'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 07:04:33.422451 8387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:33.423983 8387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:33.436403 8392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:33.436414 8395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:33.436352 8393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:33.437357 8387 server_base.cc:1034] running on GCE node
I20260621 07:04:33.438278 8387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:33.439931 8387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:33.441186 8387 hybrid_clock.cc:648] HybridClock initialized: now 1782025473441131 us; error 76 us; skew 500 ppm
I20260621 07:04:33.441578 8387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025473, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:33.444936 8387 init.cc:377] Logged in from keytab as oryx/127.1.188.253@KRBTEST.COM (short username oryx)
I20260621 07:04:33.446606 8387 webserver.cc:460] Webserver started at http://127.1.188.253:41271/ using document root <none> and password file <none>
I20260621 07:04:33.447479 8387 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:33.447607 8387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:33.447933 8387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:33.451304 8387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/instance:
uuid: "d5441c6a0617421f969b9e294261df70"
format_stamp: "Formatted at 2026-06-21 07:04:33 on dist-test-slave-hlq1"
server_key: "55c195542e23ffee40aa8f7cff8b434a"
server_key_iv: "ae92f535fba9b2eebf7733c137d48679"
server_key_version: "encryptionkey@0"
I20260621 07:04:33.452180 8387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal/instance:
uuid: "d5441c6a0617421f969b9e294261df70"
format_stamp: "Formatted at 2026-06-21 07:04:33 on dist-test-slave-hlq1"
server_key: "55c195542e23ffee40aa8f7cff8b434a"
server_key_iv: "ae92f535fba9b2eebf7733c137d48679"
server_key_version: "encryptionkey@0"
I20260621 07:04:33.457938 8387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:04:33.462076 8402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:33.463673 8387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:04:33.463901 8387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "d5441c6a0617421f969b9e294261df70"
format_stamp: "Formatted at 2026-06-21 07:04:33 on dist-test-slave-hlq1"
server_key: "55c195542e23ffee40aa8f7cff8b434a"
server_key_iv: "ae92f535fba9b2eebf7733c137d48679"
server_key_version: "encryptionkey@0"
I20260621 07:04:33.464146 8387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:33.491468 8387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:33.496423 8387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:33.496766 8387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:33.516719 8387 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.253:43365
I20260621 07:04:33.516706 8453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.253:43365 every 8 connection(s)
I20260621 07:04:33.518289 8387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
I20260621 07:04:33.520431 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8387
I20260621 07:04:33.523767 8454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:33.531131 8454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025473, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.253@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:33.545585 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.522973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.1.188.253:43365)
0621 07:04:33.523901 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.523925 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.523969 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:04:33.525950 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.525975 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.526386 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.526829 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.528395 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.529745 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.530996 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.531788 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.531973 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.536725 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.536732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.536742 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.536774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.540750 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.541405 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.541413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.541422 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.541670 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.542017 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.542023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.542026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.542796 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.543051 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.543624 (+ 573us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.544077 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":407,"threads_started":1}
I20260621 07:04:33.552285 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.533874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.1.188.254:37109)
0621 07:04:33.534428 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.534435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.534836 (+ 401us) server_negotiation.cc:407] Connection header received
0621 07:04:33.535055 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.535060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.535145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.535283 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.536491 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.537280 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.539850 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.540239 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.540335 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.544575 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.544582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.544588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.544636 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.548779 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.549783 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.549787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.549790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.549938 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.550369 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.550374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.550378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.550795 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.550978 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.551586 (+ 608us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.551848 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":251,"threads_started":1}
I20260621 07:04:33.552325 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.533883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55104)
0621 07:04:33.534693 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.534735 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.534918 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.535347 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.535352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.536083 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.536331 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.536340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.537462 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.537466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.539700 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.539717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.540437 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.540445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.540673 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.541411 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.541438 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.544316 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.548978 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.548988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.548997 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.549639 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.550059 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.550065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.550069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.550244 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.550951 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.550960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.551194 (+ 234us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.551470 (+ 276us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.551845 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":463,"threads_started":1}
I20260621 07:04:33.554461 8454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025473, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.253@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
I20260621 07:04:33.567308 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.555426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48688)
0621 07:04:33.555574 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.555589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.555685 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.555997 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.556001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.556339 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.556484 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.556492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.557597 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.557600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.558629 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.558640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.559327 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.559331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.559509 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.560150 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.560167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.562223 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.565169 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.565174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.565177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.565556 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.565902 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.565905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.565908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.566029 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.566528 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.566531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.566682 (+ 151us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.566797 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.567017 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 07:04:33.567624 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.555276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.1.188.253:43365)
0621 07:04:33.555492 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.555502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.555658 (+ 156us) server_negotiation.cc:407] Connection header received
0621 07:04:33.555789 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.555793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.555849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.555929 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.556657 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.557438 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.558755 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.559201 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.559299 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.562387 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.562392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.562395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.562424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.565019 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.565668 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.565671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.565674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.565799 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.566151 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.566155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.566157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.566424 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.566538 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.566875 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.567167 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":69}
I20260621 07:04:33.572767 8454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Bootstrap starting.
I20260621 07:04:33.576156 8454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:33.577260 8454 log.cc:824] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:33.580245 8454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: No bootstrap required, opened a new log
I20260621 07:04:33.588049 8454 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } }
I20260621 07:04:33.588508 8454 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:33.588758 8454 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5441c6a0617421f969b9e294261df70, State: Initialized, Role: FOLLOWER
I20260621 07:04:33.589545 8454 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } }
I20260621 07:04:33.590531 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } } }
I20260621 07:04:33.590895 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:33.591764 8454 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:04:33.601795 8472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:04:33.602063 8472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:04:33.604200 8387 master_runner.cc:428] Detected that this master d5441c6a0617421f969b9e294261df70 is joining an existing cluster
I20260621 07:04:33.604321 8387 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.188.253:43365
WARNING: no policy specified for oryx/127.1.188.252@KRBTEST.COM; defaulting to no policy
I20260621 07:04:33.632612 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.609838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.1.188.253:43365)
0621 07:04:33.610037 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.610045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.612885 (+ 2840us) server_negotiation.cc:407] Connection header received
0621 07:04:33.613020 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.613026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.613099 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.613217 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.614737 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.615909 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.618235 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.618992 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.619069 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.622694 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.622703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.622711 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.622750 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.626905 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.628040 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.628046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.628049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.628197 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.628631 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.628636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.628639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.629005 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.629141 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.629654 (+ 513us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.629881 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 07:04:33.632696 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.609883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48696)
0621 07:04:33.612772 (+ 2889us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.612791 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.612909 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.613263 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.613270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.614122 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.614571 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.614584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.616180 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.616186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.618081 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.618096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.619788 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.619797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.620402 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.621059 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.621086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.622490 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.627069 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.627076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.627081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.627918 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.628351 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.628356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.628360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.628517 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.629214 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.629219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.629384 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.630227 (+ 843us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.632128 (+ 1901us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2757,"mutex_wait_us":27,"thread_start_us":1710,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.252@KRBTEST.COM" created.
I20260621 07:04:33.635025 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.609160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.1.188.254:37109)
0621 07:04:33.610253 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.610259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.612065 (+ 1806us) server_negotiation.cc:407] Connection header received
0621 07:04:33.613091 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.613095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.613201 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.613299 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.614874 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.615975 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.618244 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.618908 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.619005 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.625505 (+ 6500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.625512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.625517 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.625557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.629537 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.632165 (+ 2628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.632171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.632174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.632335 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.633322 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.633327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.633330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.633738 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.633877 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.634455 (+ 578us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.634692 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963}
I20260621 07:04:33.635986 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.609222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55108)
0621 07:04:33.611946 (+ 2724us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.611967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.612190 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.613533 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.613539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.614226 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.614708 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.614721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.616181 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.616186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.618089 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.618097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.619308 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.619315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.619527 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.623923 (+ 4396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.623944 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.625208 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.629948 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.629956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.629961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.632003 (+ 2042us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.632482 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.632486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.632490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.633158 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.633957 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.633961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.634153 (+ 192us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.635196 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.635545 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2541,"mutex_wait_us":35,"thread_start_us":3181,"threads_started":1}
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025471, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.254@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
I20260621 07:04:33.661870 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.642786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.1.188.253:43365)
0621 07:04:33.642983 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.642993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.643600 (+ 607us) server_negotiation.cc:407] Connection header received
0621 07:04:33.643969 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.643977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.644116 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.644227 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.645595 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.646383 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.648327 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.649200 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.649352 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.653553 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.653562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.653570 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.653639 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.658318 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.659496 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.659506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.659511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.659698 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.660191 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.660199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.660205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.660572 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.660690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.661314 (+ 624us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.661536 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:04:33.661955 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.642729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48710)
0621 07:04:33.643474 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.643521 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.643736 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.644297 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.644304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.645117 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.645411 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.645424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.646526 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.646532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.648147 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.648167 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.649384 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.649391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.649609 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.650365 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.650389 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.653331 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.658583 (+ 5252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.658602 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.658614 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.659285 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.659838 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.659843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.659847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.660042 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.660713 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.660722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.660950 (+ 228us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.661162 (+ 212us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.661528 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":348,"threads_started":1}
I20260621 07:04:33.664404 7925 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.188.253:43365: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d5441c6a0617421f969b9e294261df70"
member_type: NON_VOTER
last_known_addr {
host: "127.1.188.253"
port: 43365
}
attrs {
promote: true
}
}
dest_uuid: "6245152a3fc4468fbbc9bd6194d309aa"
cas_config_opid_index: -1
I20260621 07:04:33.665683 7925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.252.keytab.
Entry for principal oryx/127.1.188.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.252.keytab.
I20260621 07:04:33.668892 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:41187
--webserver_interface=127.1.188.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.252.keytab
--principal=oryx/127.1.188.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:33.676293 8484 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d5441c6a0617421f969b9e294261df70 (127.1.188.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } }
I20260621 07:04:33.677539 8429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:33.680708 8429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 07:04:33.680855 8484 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.188.253:43365
I20260621 07:04:33.681632 8387 master.cc:537] Master@127.1.188.253:43365 shutting down...
I20260621 07:04:33.681668 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } }
I20260621 07:04:33.681874 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:33.683089 8485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:33.696338 8387 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:33.697005 8387 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:33.697278 8387 tablet_replica.cc:330] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: stopping tablet replica
W20260621 07:04:33.701308 7910 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 07:04:33.705624 8387 master.cc:552] Master@127.1.188.253:43365 shutdown complete.
I20260621 07:04:33.705785 8387 master_runner.cc:305] Clearing existing system tablet
I20260621 07:04:33.706308 8387 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:33.706482 8387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:33.710412 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.709944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48720)
0621 07:04:33.710104 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.710210 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 07:04:33.710875 7910 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa -> Peer d5441c6a0617421f969b9e294261df70 (127.1.188.253:43365): Couldn't send request to peer d5441c6a0617421f969b9e294261df70. Status: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:04:33.712088 8387 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:04:33.713724 8490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:33.714131 8387 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:33.714391 8387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 07:04:33.714486 8387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "d5441c6a0617421f969b9e294261df70"
format_stamp: "Formatted at 2026-06-21 07:04:33 on dist-test-slave-hlq1"
server_key: "55c195542e23ffee40aa8f7cff8b434a"
server_key_iv: "ae92f535fba9b2eebf7733c137d48679"
server_key_version: "encryptionkey@0"
I20260621 07:04:33.719712 8387 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:33.726914 8387 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:04:33.727124 8387 log.cc:1196] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 07:04:33.727638 8387 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Deleting consensus metadata
I20260621 07:04:33.727905 8387 master_runner.cc:315] Copying system tablet from 127.1.188.254:37109
I20260621 07:04:33.730536 8387 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: tablet copy: Beginning tablet copy session from remote peer at address 127.1.188.254:37109
I20260621 07:04:33.747260 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.731456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.1.188.254:37109)
0621 07:04:33.731653 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:33.731660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:33.732174 (+ 514us) server_negotiation.cc:407] Connection header received
0621 07:04:33.732383 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:33.732389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:33.732474 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:33.732583 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:33.733611 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.734793 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.736674 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.737291 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.737447 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.740474 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:33.740481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:33.740487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:33.740530 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:33.744252 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.745037 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.745043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.745047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.745208 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:33.745630 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:33.745636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:33.745640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:33.746005 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:33.746174 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:33.746655 (+ 481us) server_negotiation.cc:299] Negotiation successful
0621 07:04:33.746876 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:04:33.747292 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:33.731349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55120)
0621 07:04:33.732030 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:33.732049 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:33.732226 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:33.732631 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:33.732637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:33.733164 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:33.733439 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.733453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.734943 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.734950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.736498 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:33.736519 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:33.737751 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:33.737761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:33.738001 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:33.738838 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:33.738868 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:33.740216 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:33.744411 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.744417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.744421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.744915 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.745333 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:33.745339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:33.745344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:33.745507 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:33.746143 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:33.746147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:33.746363 (+ 216us) client_negotiation.cc:769] Sending connection context
0621 07:04:33.746524 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 07:04:33.746837 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":431,"threads_started":1}
I20260621 07:04:33.748535 7945 tablet_copy_service.cc:140] P 6245152a3fc4468fbbc9bd6194d309aa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d5441c6a0617421f969b9e294261df70 ({username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:55120)
I20260621 07:04:33.748829 7945 tablet_copy_service.cc:161] P 6245152a3fc4468fbbc9bd6194d309aa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d5441c6a0617421f969b9e294261df70 at {username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:55120: session id = d5441c6a0617421f969b9e294261df70-00000000000000000000000000000000
I20260621 07:04:33.753221 7945 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:04:33.755607 8387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:04:33.762480 8387 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: tablet copy: Starting download of 0 data blocks...
I20260621 07:04:33.762818 8387 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: tablet copy: Starting download of 1 WAL segments...
I20260621 07:04:33.765653 8387 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:04:33.770260 7945 tablet_copy_service.cc:342] P 6245152a3fc4468fbbc9bd6194d309aa: Request end of tablet copy session d5441c6a0617421f969b9e294261df70-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:55120
I20260621 07:04:33.770535 7945 tablet_copy_service.cc:433] P 6245152a3fc4468fbbc9bd6194d309aa: ending tablet copy session d5441c6a0617421f969b9e294261df70-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d5441c6a0617421f969b9e294261df70
I20260621 07:04:33.782511 8387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:33.785199 8497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:33.791669 8498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:33.795511 8500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:33.797065 8387 server_base.cc:1034] running on GCE node
I20260621 07:04:33.797487 8387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:33.798251 8387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:33.799558 8387 hybrid_clock.cc:648] HybridClock initialized: now 1782025473799489 us; error 29 us; skew 500 ppm
I20260621 07:04:33.799844 8387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:33 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025473, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:33.801919 8387 init.cc:377] Logged in from keytab as oryx/127.1.188.253@KRBTEST.COM (short username oryx)
I20260621 07:04:33.810448 8387 webserver.cc:460] Webserver started at http://127.1.188.253:33607/ using document root <none> and password file <none>
I20260621 07:04:33.810928 8387 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:33.811026 8387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:33.813611 8387 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260621 07:04:33.815603 8507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:33.816020 8387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 07:04:33.816210 8387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "d5441c6a0617421f969b9e294261df70"
format_stamp: "Formatted at 2026-06-21 07:04:33 on dist-test-slave-hlq1"
server_key: "55c195542e23ffee40aa8f7cff8b434a"
server_key_iv: "ae92f535fba9b2eebf7733c137d48679"
server_key_version: "encryptionkey@0"
I20260621 07:04:33.816377 8387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:33.845474 8387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:33.849113 8387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:33.875427 8558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.253:43365 every 8 connection(s)
I20260621 07:04:33.879607 8387 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.253:43365
I20260621 07:04:33.880681 8387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
I20260621 07:04:33.891770 8559 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:04:33.897542 8559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Bootstrap starting.
I20260621 07:04:33.911135 8381 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
I20260621 07:04:33.913127 8559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:04:33.913694 8559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Bootstrap complete.
I20260621 07:04:33.914741 8559 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } }
I20260621 07:04:33.914983 8559 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d5441c6a0617421f969b9e294261df70, State: Initialized, Role: LEARNER
I20260621 07:04:33.915256 8559 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } }
I20260621 07:04:33.922149 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } }
I20260621 07:04:33.922441 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:04:33.922983 8559 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:33.943908 8573 catalog_manager.cc:1259] Loaded cluster ID: 091540ad3b1442a7bc021c36f7783afe
I20260621 07:04:33.944042 8573 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: loading cluster ID for follower catalog manager: success
I20260621 07:04:33.946514 8573 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: acquiring CA information for follower catalog manager: success
I20260621 07:04:33.948498 8573 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:04:34.149242 8486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:34.149632 8486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:34.149778 8486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:04:34.152961 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.146141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48726)
0621 07:04:34.146406 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.146430 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.146567 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.147512 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.147517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.147547 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:34.147741 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.147751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.149341 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.149346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.150846 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.150860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.152100 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.152106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.152335 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.152403 (+ 68us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.152510 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.152664 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260621 07:04:34.153007 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.146345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.1.188.253:43365)
0621 07:04:34.147130 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.147137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.147154 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:34.147221 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.147225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.147299 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.147455 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:34.147883 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.149196 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.151000 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.151932 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.152070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.152328 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.152536 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.152663 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":520,"threads_started":1}
I20260621 07:04:34.154407 8534 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260621 07:04:34.155061 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6245152a3fc4468fbbc9bd6194d309aa. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } }
I20260621 07:04:34.155329 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: LEARNER
W20260621 07:04:34.161703 8486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:34.161823 8486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:34.161892 8486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:34.161965 8486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:34.162027 8486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:34.177902 8486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.252.keytab
--principal=oryx/127.1.188.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.188.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:34.179567 8486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:34.181252 8486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:34.194284 8580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:34.194312 8579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:34.194471 8582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:34.195030 8486 server_base.cc:1034] running on GCE node
I20260621 07:04:34.195849 8486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:34.197170 8486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:34.198401 8486 hybrid_clock.cc:648] HybridClock initialized: now 1782025474198374 us; error 56 us; skew 500 ppm
I20260621 07:04:34.198755 8486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025474, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:34.207267 8486 init.cc:377] Logged in from keytab as oryx/127.1.188.252@KRBTEST.COM (short username oryx)
I20260621 07:04:34.208834 8486 webserver.cc:460] Webserver started at http://127.1.188.252:40309/ using document root <none> and password file <none>
I20260621 07:04:34.209601 8486 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:34.209745 8486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:34.210047 8486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:34.213086 8486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/instance:
uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
format_stamp: "Formatted at 2026-06-21 07:04:34 on dist-test-slave-hlq1"
server_key: "1b983fb4c4309df15c96cc5ac0dcc7bd"
server_key_iv: "78f8112ae744d3ce2eaa128e06f81af8"
server_key_version: "encryptionkey@0"
I20260621 07:04:34.213872 8486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal/instance:
uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
format_stamp: "Formatted at 2026-06-21 07:04:34 on dist-test-slave-hlq1"
server_key: "1b983fb4c4309df15c96cc5ac0dcc7bd"
server_key_iv: "78f8112ae744d3ce2eaa128e06f81af8"
server_key_version: "encryptionkey@0"
I20260621 07:04:34.219316 8486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:04:34.222759 8589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:34.224187 8486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:04:34.224376 8486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
format_stamp: "Formatted at 2026-06-21 07:04:34 on dist-test-slave-hlq1"
server_key: "1b983fb4c4309df15c96cc5ac0dcc7bd"
server_key_iv: "78f8112ae744d3ce2eaa128e06f81af8"
server_key_version: "encryptionkey@0"
I20260621 07:04:34.224570 8486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:34.248394 8486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:34.252445 8486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:34.252730 8486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:34.270313 8486 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.252:41187
I20260621 07:04:34.270335 8640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.252:41187 every 8 connection(s)
I20260621 07:04:34.271747 8486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
I20260621 07:04:34.275336 8641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:34.278461 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8486
I20260621 07:04:34.283006 8641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025474, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.252@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:34.301386 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.285795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.1.188.254:37109)
0621 07:04:34.286995 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.287001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.287020 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:04:34.287077 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.287082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.287160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.287285 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.288461 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.289140 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.290620 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.291343 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.291517 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.295155 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.295160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.295163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.295195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.298179 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.298957 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.298963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.298966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.299188 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.299698 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.299701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.299704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.299980 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.300178 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.300798 (+ 620us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.301025 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":996,"threads_started":1}
I20260621 07:04:34.302536 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.280806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.1.188.252:41187)
0621 07:04:34.281528 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.281539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.281580 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:04:34.283350 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.283371 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.283826 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.284195 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.285749 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.287721 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.289020 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.289679 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.289889 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.294594 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.294605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.294621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.294663 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.297818 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.298621 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.298629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.298633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.298828 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.299293 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.299298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.299301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.299954 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.300236 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.300618 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.301153 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":538,"thread_start_us":346,"threads_started":1}
I20260621 07:04:34.302558 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.285619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55128)
0621 07:04:34.286558 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.286621 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.286786 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.287333 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.287338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.288029 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.288303 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.288312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.289287 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.289298 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.290487 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.290503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.291561 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.291566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.291693 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.292282 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.292304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.294914 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.298310 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.298319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.298328 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.298828 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.299302 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.299306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.299310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.299546 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.300173 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.300182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.300373 (+ 191us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.300678 (+ 305us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.301250 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":193,"spinlock_wait_cycles":22784,"thread_start_us":399,"threads_started":1}
I20260621 07:04:34.304261 8641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025474, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.252@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
I20260621 07:04:34.318428 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.305086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48738)
0621 07:04:34.305282 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.305302 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.305432 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.305777 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.305780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.306103 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.306233 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.306241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.307206 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.307209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.308210 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.308221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.309000 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.309004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.309205 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.309872 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.309892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.312370 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.315637 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.315642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.315647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.316071 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.316617 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.316623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.316630 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.316850 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.317529 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.317534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.317730 (+ 196us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.317918 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.318174 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 07:04:34.318673 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.305026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.1.188.253:43365)
0621 07:04:34.305245 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.305251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.305413 (+ 162us) server_negotiation.cc:407] Connection header received
0621 07:04:34.305543 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.305550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.305656 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.305737 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.306377 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.307075 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.308395 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.308850 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.308964 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.312582 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.312587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.312591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.312625 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.315510 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.316299 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.316306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.316311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.316459 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.316977 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.316982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.316986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.317310 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.317573 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.318038 (+ 465us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.318311 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260621 07:04:34.320858 8641 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025474, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.252@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
I20260621 07:04:34.333766 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.321561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57618)
0621 07:04:34.321744 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.321762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.321860 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.322155 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.322158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.322502 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.322702 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.322711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.323819 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.323823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.324785 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.324796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.325466 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.325471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.325631 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.326208 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.326224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.328389 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.331350 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.331354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.331357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.331822 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.332229 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.332233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.332235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.332393 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.332971 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.332975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.333094 (+ 119us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.333230 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.333434 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:04:34.333796 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.321680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.1.188.252:41187)
0621 07:04:34.321840 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.321844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.321861 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:34.321955 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.321959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.322013 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.322094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.322836 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.323671 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.324908 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.325334 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.325438 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.328553 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.328558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.328561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.328588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.331226 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.331930 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.331934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.331936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.332120 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.332534 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.332541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.332546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.332872 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.332977 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.333312 (+ 335us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.333559 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":62}
I20260621 07:04:34.339111 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Bootstrap starting.
I20260621 07:04:34.342445 8641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:34.343581 8641 log.cc:824] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:34.346243 8641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: No bootstrap required, opened a new log
I20260621 07:04:34.354445 8641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } }
I20260621 07:04:34.354876 8641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:34.355023 8641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5acc76e6da4ec9bc00c7f1f629c578, State: Initialized, Role: FOLLOWER
I20260621 07:04:34.355729 8641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } }
I20260621 07:04:34.356810 8648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } } }
I20260621 07:04:34.357271 8648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:34.358255 8641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:04:34.367712 8659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:04:34.367888 8659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:04:34.372602 8486 master_runner.cc:428] Detected that this master 9d5acc76e6da4ec9bc00c7f1f629c578 is joining an existing cluster
I20260621 07:04:34.372782 8486 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.188.252:41187
I20260621 07:04:34.397758 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.376899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.1.188.253:43365)
0621 07:04:34.377059 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.377064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.377708 (+ 644us) server_negotiation.cc:407] Connection header received
0621 07:04:34.378015 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.378024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.378138 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.378222 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.380170 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.381596 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.384418 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.385344 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.385465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.389128 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.389137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.389145 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.389197 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.393652 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.394701 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.394709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.394716 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.394921 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.395581 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.395589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.395595 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.396010 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.396153 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.396948 (+ 795us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.397301 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:04:34.398989 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.376898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48740)
0621 07:04:34.377591 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.377614 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.377844 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.378498 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.378506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.379455 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.379939 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.379955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.381828 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.381837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.384184 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.384205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.385861 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.385870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.386145 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.387138 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.387170 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.388843 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.393879 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.393887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.393894 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.394535 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.395120 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.395128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.395134 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.395428 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.396231 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.396238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.396561 (+ 323us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.398093 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.398448 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":372,"threads_started":1}
I20260621 07:04:34.405368 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.376766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.1.188.254:37109)
0621 07:04:34.376979 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.376987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.388111 (+ 11124us) server_negotiation.cc:407] Connection header received
0621 07:04:34.388608 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.388614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.388717 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.388801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.390381 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.391693 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.393941 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.394665 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.394770 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.398939 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.398947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.398953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.399003 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.402247 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.402990 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.402996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.403000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.403149 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.403692 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.403697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.403701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.404092 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.404271 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.404768 (+ 497us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.404996 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:04:34.405537 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.377415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55144)
0621 07:04:34.387973 (+ 10558us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.388000 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.388405 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.389049 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.389054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.389813 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.390194 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.390207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.391906 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.391913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.393762 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.393778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.395092 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.395099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.395378 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.396620 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.396638 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.398693 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.402392 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.402399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.402404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.402837 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.403300 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.403305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.403309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.403550 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.404235 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.404242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.404461 (+ 219us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.404643 (+ 182us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.404982 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10370,"mutex_wait_us":49,"spinlock_wait_cycles":896,"thread_start_us":18474,"threads_started":1}
I20260621 07:04:34.409106 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7986
I20260621 07:04:34.413760 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.378198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.1.188.252:41187)
0621 07:04:34.378393 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.378400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.396379 (+ 17979us) server_negotiation.cc:407] Connection header received
0621 07:04:34.396737 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.396748 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.396835 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.396901 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.398726 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.399871 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.402851 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.403676 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.403814 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.406270 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.406276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.406280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.406315 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.410157 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.411017 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.411024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.411029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.411209 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.411729 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.411737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.411743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.412097 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.412273 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.412944 (+ 671us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.413237 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:04:34.415390 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.377746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57632)
0621 07:04:34.396284 (+ 18538us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.396306 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.396571 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.397092 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.397097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.398224 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.398555 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.398568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.400290 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.400298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.402689 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.402705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.403865 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.403871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.404042 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.404786 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.404830 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.406072 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.410302 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.410306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.410310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.410850 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.411381 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.411389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.411415 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.411598 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.412360 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.412364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.412602 (+ 238us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.414297 (+ 1695us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.414741 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18375,"mutex_wait_us":1,"thread_start_us":10413,"threads_started":1}
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025471, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.254@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
I20260621 07:04:34.429031 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.411649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57638)
0621 07:04:34.411968 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.411986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.412086 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.413029 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.413034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.413465 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.413677 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.413689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.415880 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.415886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.417283 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.417298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.418415 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.418424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.418665 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.419304 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.419329 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.422276 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.426344 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.426349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.426353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.426697 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.427161 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.427166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.427168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.427301 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.427980 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.427985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.428161 (+ 176us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.428305 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.428628 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260621 07:04:34.429098 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.411941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.1.188.252:41187)
0621 07:04:34.412649 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.412654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.412680 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:34.412768 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.412773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.412854 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.413861 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.414237 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.415729 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.417522 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.418231 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.418364 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.422526 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.422533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.422539 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.422587 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.426212 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.426812 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.426819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.426822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.426964 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.427488 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.427493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.427496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.427860 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.428012 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.428450 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.428705 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":390,"threads_started":1}
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.193@KRBTEST.COM
I20260621 07:04:34.430660 7925 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.188.252:41187: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
member_type: NON_VOTER
last_known_addr {
host: "127.1.188.252"
port: 41187
}
attrs {
promote: true
}
}
dest_uuid: "6245152a3fc4468fbbc9bd6194d309aa"
cas_config_opid_index: 5
I20260621 07:04:34.431675 7925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } }
I20260621 07:04:34.434482 8616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:34.434520 8534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:04:34.435365 8670 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.438325 8616 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 07:04:34.438918 8485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 07:04:34.440060 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.410881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.1.188.193:38565)
0621 07:04:34.418983 (+ 8102us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.418993 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.419026 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:04:34.419100 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.419110 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.419427 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.419607 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.421322 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.422425 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.424408 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.425339 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.425473 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.431114 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.431130 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.431151 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.431226 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.435510 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.436553 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.436562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.436566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.436756 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.437210 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.437216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.437219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.437629 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.439044 (+ 1415us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.439281 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.439598 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7907,"thread_start_us":7737,"threads_started":1}
I20260621 07:04:34.441938 8534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.444972 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.445245 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:04:34.446202 8671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:34.446681 8670 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.188.252:41187
I20260621 07:04:34.446838 8109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:34.447481 8486 master.cc:537] Master@127.1.188.252:41187 shutting down...
I20260621 07:04:34.447281 8485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.447706 8485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:34.453708 8670 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: attempting to promote NON_VOTER d5441c6a0617421f969b9e294261df70 to VOTER
I20260621 07:04:34.466823 8670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } }
I20260621 07:04:34.469182 8485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: NON_VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: true } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.469614 8485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:34.469969 8534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 LEARNER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260621 07:04:34.470722 8673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260621 07:04:34.473559 8485 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d5441c6a0617421f969b9e294261df70 (127.1.188.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.477852 8670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.478133 8485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.478410 8485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:34.478427 8670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:34.479748 8534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d5441c6a0617421f969b9e294261df70 (127.1.188.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.483805 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.484059 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:34.489823 8486 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:34.490272 8486 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:34.490483 8486 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: stopping tablet replica
I20260621 07:04:34.497982 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.497475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57648)
0621 07:04:34.497673 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.497777 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 07:04:34.498353 7910 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa -> Peer 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252:41187): Couldn't send request to peer 9d5acc76e6da4ec9bc00c7f1f629c578. Status: Network error: Client connection negotiation failed: client connection to 127.1.188.252:41187: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:04:34.508383 8486 master.cc:552] Master@127.1.188.252:41187 shutdown complete.
I20260621 07:04:34.508481 8486 master_runner.cc:305] Clearing existing system tablet
I20260621 07:04:34.508899 8486 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:34.508996 8486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:34.512085 8486 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:04:34.513695 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:34.514032 8486 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:34.514262 8486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260621 07:04:34.514354 8486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
format_stamp: "Formatted at 2026-06-21 07:04:34 on dist-test-slave-hlq1"
server_key: "1b983fb4c4309df15c96cc5ac0dcc7bd"
server_key_iv: "78f8112ae744d3ce2eaa128e06f81af8"
server_key_version: "encryptionkey@0"
I20260621 07:04:34.519505 8486 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:34.526588 8486 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 07:04:34.526741 8486 log.cc:1196] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 07:04:34.527184 8486 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Deleting consensus metadata
I20260621 07:04:34.527448 8486 master_runner.cc:315] Copying system tablet from 127.1.188.254:37109
I20260621 07:04:34.533082 8486 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: tablet copy: Beginning tablet copy session from remote peer at address 127.1.188.254:37109
I20260621 07:04:34.549288 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.533892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55156)
0621 07:04:34.534627 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.534654 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.534802 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.535204 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.535209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.535819 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:34.536092 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.536104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.537877 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.537883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.539510 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.539525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.540527 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.540532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.540720 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.541458 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:34.541480 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:34.542547 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:34.546443 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.546449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.546454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.546971 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.547388 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:34.547393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:34.547417 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:34.547576 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:34.548214 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:34.548218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:34.548414 (+ 196us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.548567 (+ 153us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.548862 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260621 07:04:34.549322 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.534209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.1.188.254:37109)
0621 07:04:34.534400 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.534406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.534766 (+ 360us) server_negotiation.cc:407] Connection header received
0621 07:04:34.534952 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.534959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.535044 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.535158 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:34.536254 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.537703 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.539671 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.540353 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.540499 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.542770 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:34.542776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:34.542782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:34.542831 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:34.546294 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.547097 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.547102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.547106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.547264 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:34.547700 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:34.547707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:34.547711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:34.548086 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:34.548235 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.548718 (+ 483us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.548950 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:04:34.550437 7945 tablet_copy_service.cc:140] P 6245152a3fc4468fbbc9bd6194d309aa: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d5acc76e6da4ec9bc00c7f1f629c578 ({username='oryx', principal='oryx/127.1.188.252@KRBTEST.COM'} at 127.0.0.1:55156)
I20260621 07:04:34.550616 7945 tablet_copy_service.cc:161] P 6245152a3fc4468fbbc9bd6194d309aa: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d5acc76e6da4ec9bc00c7f1f629c578 at {username='oryx', principal='oryx/127.1.188.252@KRBTEST.COM'} at 127.0.0.1:55156: session id = 9d5acc76e6da4ec9bc00c7f1f629c578-00000000000000000000000000000000
I20260621 07:04:34.554286 7945 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 07:04:34.556275 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 07:04:34.562948 8486 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: tablet copy: Starting download of 0 data blocks...
I20260621 07:04:34.563236 8486 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: tablet copy: Starting download of 1 WAL segments...
I20260621 07:04:34.565639 8486 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 07:04:34.570031 7945 tablet_copy_service.cc:342] P 6245152a3fc4468fbbc9bd6194d309aa: Request end of tablet copy session 9d5acc76e6da4ec9bc00c7f1f629c578-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.188.252@KRBTEST.COM'} at 127.0.0.1:55156
I20260621 07:04:34.570173 7945 tablet_copy_service.cc:433] P 6245152a3fc4468fbbc9bd6194d309aa: ending tablet copy session 9d5acc76e6da4ec9bc00c7f1f629c578-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d5acc76e6da4ec9bc00c7f1f629c578
I20260621 07:04:34.578938 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7986
I20260621 07:04:34.584263 8486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:34.591039 8685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:34.596141 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:34.605397 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:34.605865 8486 server_base.cc:1034] running on GCE node
I20260621 07:04:34.606251 8486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:34.606966 8486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:34.608268 8486 hybrid_clock.cc:648] HybridClock initialized: now 1782025474608181 us; error 75 us; skew 500 ppm
I20260621 07:04:34.608534 8486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:04:34.608536 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:38565
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:37109
--webserver_port=41625
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:04:34 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025474, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:34.610761 8486 init.cc:377] Logged in from keytab as oryx/127.1.188.252@KRBTEST.COM (short username oryx)
I20260621 07:04:34.615535 8486 webserver.cc:460] Webserver started at http://127.1.188.252:37683/ using document root <none> and password file <none>
I20260621 07:04:34.615963 8486 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:34.616068 8486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:34.619496 8486 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260621 07:04:34.621474 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:34.622012 8486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260621 07:04:34.622130 8486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
format_stamp: "Formatted at 2026-06-21 07:04:34 on dist-test-slave-hlq1"
server_key: "1b983fb4c4309df15c96cc5ac0dcc7bd"
server_key_iv: "78f8112ae744d3ce2eaa128e06f81af8"
server_key_version: "encryptionkey@0"
I20260621 07:04:34.622354 8486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:34.654407 8486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:34.658453 8486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:34.687495 8486 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.252:41187
I20260621 07:04:34.687536 8747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.252:41187 every 8 connection(s)
I20260621 07:04:34.688545 8486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
I20260621 07:04:34.694219 8748 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:04:34.698328 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Bootstrap starting.
I20260621 07:04:34.718307 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:04:34.718866 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: Bootstrap complete.
I20260621 07:04:34.720026 8748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } }
I20260621 07:04:34.720356 8748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d5acc76e6da4ec9bc00c7f1f629c578, State: Initialized, Role: LEARNER
I20260621 07:04:34.720626 8748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } }
I20260621 07:04:34.721558 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.721849 8748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:34.722018 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: LEARNER
I20260621 07:04:34.738356 8762 catalog_manager.cc:1259] Loaded cluster ID: 091540ad3b1442a7bc021c36f7783afe
I20260621 07:04:34.738478 8762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: loading cluster ID for follower catalog manager: success
I20260621 07:04:34.742730 8762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: acquiring CA information for follower catalog manager: success
I20260621 07:04:34.745450 8762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:04:34.971232 8693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:34.971638 8693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:34.971802 8693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:04:34.974246 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.967485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57662)
0621 07:04:34.967680 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:34.967701 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:34.967833 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:34.968477 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:34.968482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:34.968511 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:34.968691 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.968701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.969895 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.969901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.971847 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:34.971864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.973333 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.973339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:34.973540 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.973574 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:04:34.973701 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 07:04:34.973854 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:04:34.974241 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:34.967611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.1.188.252:41187)
0621 07:04:34.968165 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:34.968170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:34.968187 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:34.968249 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:34.968252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:34.968320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:34.968421 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:34.968824 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.969761 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.972027 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:34.973108 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:34.973301 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:34.973504 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:34.973725 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 07:04:34.973853 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":309,"threads_started":1}
I20260621 07:04:34.975137 8723 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260621 07:04:34.975874 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6245152a3fc4468fbbc9bd6194d309aa. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:34.976212 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: LEARNER
W20260621 07:04:34.980609 8693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:34.980715 8693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:34.980840 8693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:34.990765 8693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:38565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=41625
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:34.992118 8693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:34.993364 8693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:35.007695 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:35.007701 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:35.007938 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:35.008394 8693 server_base.cc:1034] running on GCE node
I20260621 07:04:35.009114 8693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:35.010160 8693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:35.011505 8693 hybrid_clock.cc:648] HybridClock initialized: now 1782025475011442 us; error 42 us; skew 500 ppm
I20260621 07:04:35.011863 8693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:35.015255 8693 init.cc:377] Logged in from keytab as oryx/127.1.188.193@KRBTEST.COM (short username oryx)
I20260621 07:04:35.017066 8693 webserver.cc:460] Webserver started at http://127.1.188.193:41625/ using document root <none> and password file <none>
I20260621 07:04:35.017802 8693 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:35.017915 8693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:35.023784 8693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:04:35.026446 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.027792 8693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:35.027959 8693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "e68f9669d85a4a36bbbc17470fbe03fc"
format_stamp: "Formatted at 2026-06-21 07:04:31 on dist-test-slave-hlq1"
server_key: "3b3012c01be447dea925715fd025af60"
server_key_iv: "2dc7a7a5e2ad400d8a259fbebb15701a"
server_key_version: "encryptionkey@0"
I20260621 07:04:35.028915 8693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:35.047472 8693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:35.051959 8693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:35.052237 8693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:35.053210 8693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:35.054620 8693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:35.054704 8693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.054858 8693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:35.054935 8693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.088507 8693 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:38565
I20260621 07:04:35.088564 8890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:38565 every 8 connection(s)
I20260621 07:04:35.089872 8693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:35.093103 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8693
I20260621 07:04:35.093504 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8120
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.194@KRBTEST.COM
I20260621 07:04:35.137105 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.108237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:53045 (local address 127.1.188.252:41187)
0621 07:04:35.108467 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.108475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.116108 (+ 7633us) server_negotiation.cc:407] Connection header received
0621 07:04:35.116606 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.116615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.116706 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.116782 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.118498 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.119666 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.123141 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.123921 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.124037 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.129438 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.129443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.129446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.129479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.133095 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.134123 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.134131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.134135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.134301 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.134849 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.134857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.134862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.135231 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.135356 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.136426 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.136681 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:04:35.139125 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.108657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.1.188.193:53045)
0621 07:04:35.115975 (+ 7318us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.115992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.116437 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.117002 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.117007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.117945 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.118340 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.118349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.119858 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.119864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.122853 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.122869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.124422 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.124429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.125018 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.125747 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.125772 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.129178 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.133284 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.133296 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.133318 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.133952 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.134461 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.134466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.134469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.134721 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.135488 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.135497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.135952 (+ 455us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.137620 (+ 1668us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.137883 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7192,"mutex_wait_us":738,"thread_start_us":7462,"threads_started":1}
I20260621 07:04:35.139894 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.092666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:52855 (local address 127.1.188.253:43365)
0621 07:04:35.115282 (+ 22616us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.115291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.115325 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:04:35.115524 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.115533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.115651 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.117275 (+ 1624us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.118494 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.119666 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.122749 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.123444 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.123583 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.130158 (+ 6575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.130167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.130173 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.130220 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.134656 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.135497 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.135502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.135505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.135654 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.136084 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.136089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.136092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.136441 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.136572 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.139251 (+ 2679us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.139517 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22445,"thread_start_us":22714,"threads_started":1}
I20260621 07:04:35.140031 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.092735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:32939 (local address 127.1.188.254:37109)
0621 07:04:35.121942 (+ 29207us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.121949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.121969 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:04:35.122027 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.122035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.122153 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.122276 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.123716 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.124718 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.126756 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.128237 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.128413 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.132477 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.132487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.132493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.132541 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.136406 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.137207 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.137224 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.137228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.137369 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.137873 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.137880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.137888 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.138263 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.139251 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.139427 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.139659 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29041,"thread_start_us":28944,"threads_started":1}
I20260621 07:04:35.139092 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.093928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.1.188.193:52855)
0621 07:04:35.107853 (+ 13925us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.107874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.112320 (+ 4446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.116167 (+ 3847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.116179 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.117278 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.118266 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.118300 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.119858 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.119864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.122445 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.122460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.123906 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.123913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.124192 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.125013 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.125047 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.129210 (+ 4163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.134808 (+ 5598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.134815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.134822 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.135326 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.135783 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.135787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.135790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.135947 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.136650 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.136654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.136801 (+ 147us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.136981 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.137546 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13798,"mutex_wait_us":72,"thread_start_us":13617,"threads_started":1}
I20260621 07:04:35.139873 8897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.093336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:32939)
0621 07:04:35.105505 (+ 12169us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.105566 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.114485 (+ 8919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.122538 (+ 8053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.122544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.123171 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.123443 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.123508 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.124921 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.124928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.126568 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.126583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.128490 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.128496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.128699 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.129443 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.129463 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.132052 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.136561 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.136569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.136575 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.137040 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.137500 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.137507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.137510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.137715 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.138405 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.138412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.138557 (+ 145us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.138707 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.139062 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11816,"mutex_wait_us":2,"thread_start_us":11553,"threads_started":1}
I20260621 07:04:35.141618 8891 heartbeater.cc:347] Connected to a master server at 127.1.188.252:41187
I20260621 07:04:35.141680 8892 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:35.141783 8893 heartbeater.cc:347] Connected to a master server at 127.1.188.253:43365
I20260621 07:04:35.141992 8892 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.141987 8891 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.141987 8893 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.143071 8892 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:35.143188 8893 heartbeater.cc:511] Master 127.1.188.253:43365 requested a full tablet report, sending...
I20260621 07:04:35.143570 8891 heartbeater.cc:511] Master 127.1.188.252:41187 requested a full tablet report, sending...
I20260621 07:04:35.144956 7925 ts_manager.cc:194] Re-registered known tserver with Master: e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:35.145252 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.095197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.1.188.194:43957)
0621 07:04:35.121318 (+ 26121us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.121331 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.121372 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:04:35.121459 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.121472 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.121808 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.121974 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.124526 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.127259 (+ 2733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.129262 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.130201 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.130329 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.136793 (+ 6464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.136811 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.136830 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.136878 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.141338 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.142286 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.142295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.142299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.142472 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.142954 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.142959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.142962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.143391 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.143730 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.144567 (+ 837us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.144832 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25889,"thread_start_us":26175,"threads_started":1}
I20260621 07:04:35.145550 8524 ts_manager.cc:194] Registered new tserver with Master: e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:35.145859 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:32939
I20260621 07:04:35.147315 8713 ts_manager.cc:194] Registered new tserver with Master: e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:35.149204 8243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:35.232748 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8120
I20260621 07:04:35.249848 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:43957
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:37109
--webserver_port=40247
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:35.346973 8673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: NON_VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: true } } }
I20260621 07:04:35.347352 8673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:35.349067 8904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: attempting to promote NON_VOTER 9d5acc76e6da4ec9bc00c7f1f629c578 to VOTER
I20260621 07:04:35.349678 8904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:35.351591 8723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 LEARNER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 07:04:35.351624 8534 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Refusing update from remote peer 6245152a3fc4468fbbc9bd6194d309aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 07:04:35.352178 8673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 07:04:35.352622 8904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 07:04:35.354956 8673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.356702 8723 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.359103 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.359445 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:35.359726 8673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.359941 8673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:35.360005 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.360248 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:35.360297 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.360502 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:35.361038 8534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.365276 8908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6245152a3fc4468fbbc9bd6194d309aa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:35.365667 8908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 07:04:35.581923 8903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:35.582202 8903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:35.582356 8903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:35.589680 8903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:35.589805 8903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:35.589985 8903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:35.600063 8903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:43957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=40247
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:35.601575 8903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:35.602843 8903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:35.616757 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:35.616797 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:35.616950 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:35.618192 8903 server_base.cc:1034] running on GCE node
I20260621 07:04:35.618893 8903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:35.620021 8903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:35.621279 8903 hybrid_clock.cc:648] HybridClock initialized: now 1782025475621215 us; error 52 us; skew 500 ppm
I20260621 07:04:35.621766 8903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:35.626652 8903 init.cc:377] Logged in from keytab as oryx/127.1.188.194@KRBTEST.COM (short username oryx)
I20260621 07:04:35.628374 8903 webserver.cc:460] Webserver started at http://127.1.188.194:40247/ using document root <none> and password file <none>
I20260621 07:04:35.629122 8903 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:35.629278 8903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:35.635208 8903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:04:35.638230 8923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.639698 8903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:04:35.639883 8903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "ecf10dfd450b4ed48fe793db0ed11988"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "5d2683d33a502c63548624e145000036"
server_key_iv: "70a36eea89578474c240fc58b71fdc1b"
server_key_version: "encryptionkey@0"
I20260621 07:04:35.640980 8903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:35.662470 8903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:35.666611 8903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:35.666895 8903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:35.667905 8903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:35.669313 8903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:35.669400 8903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.669554 8903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:35.669633 8903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:35.703686 8903 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43957
I20260621 07:04:35.703776 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43957 every 8 connection(s)
I20260621 07:04:35.705152 8903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:35.711879 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 8903
I20260621 07:04:35.712178 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8253
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:35.736331 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.708070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:43937 (local address 127.1.188.252:41187)
0621 07:04:35.709673 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.709680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.710360 (+ 680us) server_negotiation.cc:407] Connection header received
0621 07:04:35.713229 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.713235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.713322 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.713405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.716176 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.717635 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.720715 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.721517 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.721632 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.728359 (+ 6727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.728366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.728371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.728415 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.732454 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.733533 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.733539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.733543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.733730 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.734248 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.734253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.734256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.734515 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.734677 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.735637 (+ 960us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.735873 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":1296,"threads_started":1}
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025475, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
Jun 21 07:04:35 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.195@KRBTEST.COM
I20260621 07:04:35.738775 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.708699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.1.188.194:43937)
0621 07:04:35.710267 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.710285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.712865 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.714008 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.714022 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.714941 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.715936 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.715964 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.717922 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.717930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.720520 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.720538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.722051 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.722060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.722441 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.723338 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.723368 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.728060 (+ 4692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.732663 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.732672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.732728 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.733387 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.733879 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.733886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.733890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.734152 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.734654 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.734662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.735133 (+ 471us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.736634 (+ 1501us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.737192 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"mutex_wait_us":321,"spinlock_wait_cycles":6400,"thread_start_us":927,"threads_started":1}
I20260621 07:04:35.740772 9036 heartbeater.cc:347] Connected to a master server at 127.1.188.252:41187
I20260621 07:04:35.741048 9036 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.742019 9036 heartbeater.cc:511] Master 127.1.188.252:41187 requested a full tablet report, sending...
I20260621 07:04:35.742144 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.708318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55043 (local address 127.1.188.254:37109)
0621 07:04:35.709233 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.709245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.710360 (+ 1115us) server_negotiation.cc:407] Connection header received
0621 07:04:35.722773 (+ 12413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.722780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.723237 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.723323 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.725863 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.726955 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.729054 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.729744 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.729851 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.734261 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.734269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.734276 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.734321 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.738264 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.739854 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.739861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.739867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.740026 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.740462 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.740467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.740470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.740816 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.740959 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.741501 (+ 542us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.741724 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":589,"threads_started":1}
I20260621 07:04:35.743063 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.708656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:55043)
0621 07:04:35.710177 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.710230 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.722585 (+ 12355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.723732 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.723742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.724372 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.724712 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.724726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.727230 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.727234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.728881 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.728896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.730132 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.730144 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.730396 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.731035 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.731059 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.733983 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.738485 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.738492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.738499 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.739705 (+ 1206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.740193 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.740197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.740200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.740339 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.741046 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.741052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.741227 (+ 175us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.742320 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.742619 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":3,"spinlock_wait_cycles":11648,"thread_start_us":816,"threads_started":1}
I20260621 07:04:35.743925 9037 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:35.744046 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.710902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.1.188.194:33775)
0621 07:04:35.726920 (+ 16018us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:35.726940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:35.727049 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:35.727761 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:35.727766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:35.728439 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:35.728796 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.728809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.730209 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.730216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.731900 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:35.731913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.733272 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.733278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:35.733557 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.734808 (+ 1251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:35.734827 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:35.737422 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:35.741259 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.741263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.741267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.741761 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.742190 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:35.742195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:35.742198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:35.742357 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:35.743027 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:35.743034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:35.743202 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 07:04:35.743331 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 07:04:35.743623 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15875,"thread_start_us":15764,"threads_started":1}
I20260621 07:04:35.744077 9037 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.744535 9037 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:35.744652 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.711033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:33775 (local address 127.1.188.253:43365)
0621 07:04:35.727248 (+ 16215us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.727254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.727273 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:04:35.727332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.727336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.727457 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.727567 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.728982 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.730045 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.732144 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.732846 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.732952 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.737646 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.737653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.737658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.737712 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.741114 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.741902 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.741910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.741915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.742050 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.742478 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.742483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.742486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.742815 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.742961 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.744133 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.744367 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16099,"thread_start_us":15980,"threads_started":1}
I20260621 07:04:35.745378 8713 ts_manager.cc:194] Registered new tserver with Master: ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957)
I20260621 07:04:35.745591 9038 heartbeater.cc:347] Connected to a master server at 127.1.188.253:43365
I20260621 07:04:35.745544 7925 ts_manager.cc:194] Re-registered known tserver with Master: ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957)
I20260621 07:04:35.745689 9038 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:35.746076 9038 heartbeater.cc:511] Master 127.1.188.253:43365 requested a full tablet report, sending...
I20260621 07:04:35.746701 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:55043
I20260621 07:04:35.747244 8524 ts_manager.cc:194] Registered new tserver with Master: ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957)
I20260621 07:04:35.748845 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:35.713343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.1.188.195:37761)
0621 07:04:35.723141 (+ 9798us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:35.723154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:35.723632 (+ 478us) server_negotiation.cc:407] Connection header received
0621 07:04:35.723729 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:35.723741 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:35.724060 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:35.724231 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:35.728202 (+ 3971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.729841 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.731829 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:35.732687 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:35.732813 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:35.740047 (+ 7234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:35.740064 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:35.740080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:35.740141 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:35.744728 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.746094 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.746102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.746106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.746248 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:35.746689 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:35.746694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:35.746697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:35.747632 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:35.747954 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:35.748173 (+ 219us) server_negotiation.cc:299] Negotiation successful
0621 07:04:35.748455 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9574,"thread_start_us":8950,"threads_started":1}
I20260621 07:04:35.750146 8376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:35.836273 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8253
I20260621 07:04:35.853372 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:37761
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:37109
--webserver_port=37289
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:36.148715 8892 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
W20260621 07:04:36.202906 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:36.203214 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:36.203379 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:36.210639 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:36.210789 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:36.211021 9049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:36.221186 9049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:37761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=37289
--tserver_master_addrs=127.1.188.254:37109,127.1.188.253:43365,127.1.188.252:41187
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:36.222591 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:36.224211 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:36.237370 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:36.237468 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:36.237411 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:36.238193 9049 server_base.cc:1034] running on GCE node
I20260621 07:04:36.238809 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:36.239859 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:36.241128 9049 hybrid_clock.cc:648] HybridClock initialized: now 1782025476241069 us; error 45 us; skew 500 ppm
I20260621 07:04:36.241516 9049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:36 dist-test-slave-hlq1 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:36.245056 9049 init.cc:377] Logged in from keytab as oryx/127.1.188.195@KRBTEST.COM (short username oryx)
I20260621 07:04:36.246548 9049 webserver.cc:460] Webserver started at http://127.1.188.195:37289/ using document root <none> and password file <none>
I20260621 07:04:36.247248 9049 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:36.247465 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:36.252918 9049 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:36.255779 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:36.257299 9049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:36.257467 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "5d753a88e97545f1958c842e6c28012a"
format_stamp: "Formatted at 2026-06-21 07:04:32 on dist-test-slave-hlq1"
server_key: "68bcff1284933cfbb70a0d8886cf9de8"
server_key_iv: "17ea62f80cb042272bc1cfa30e0b4d66"
server_key_version: "encryptionkey@0"
I20260621 07:04:36.258492 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:36.288754 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:36.293041 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:36.293337 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:36.294322 9049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:36.295761 9049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:36.295885 9049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:36.296013 9049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:36.296113 9049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:36.333823 9049 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:37761
I20260621 07:04:36.333832 9176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:37761 every 8 connection(s)
I20260621 07:04:36.335464 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:36.337879 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 9049
Jun 21 07:04:36 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.252@KRBTEST.COM
I20260621 07:04:36.402235 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.339778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:36943 (local address 127.1.188.252:41187)
0621 07:04:36.343713 (+ 3935us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.343718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.355150 (+ 11432us) server_negotiation.cc:407] Connection header received
0621 07:04:36.368663 (+ 13513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.368674 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.368812 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.370098 (+ 1286us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.376665 (+ 6567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.378191 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.381832 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.382944 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.383075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.389661 (+ 6586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.389672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.389681 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.389737 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.394536 (+ 4799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.395897 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.395906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.395912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.396094 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.396760 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.396768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.396774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.397230 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.397397 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.401546 (+ 4149us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.401846 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3796,"thread_start_us":3672,"threads_started":1}
I20260621 07:04:36.405695 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.339126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.1.188.195:36943)
0621 07:04:36.354876 (+ 15750us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.354954 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.366607 (+ 11653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.371021 (+ 4414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.371040 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.372208 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:36.373297 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.373330 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.378642 (+ 5312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.378650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.381600 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.381623 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.383553 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.383573 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.384012 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.385069 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:36.385112 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:36.389241 (+ 4129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:36.394798 (+ 5557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.394811 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.394837 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.395717 (+ 880us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.396279 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.396286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.396292 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.396600 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.397514 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:36.400238 (+ 2724us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:36.400945 (+ 707us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.402593 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.403351 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15228,"thread_start_us":15966,"threads_started":1}
I20260621 07:04:36.415057 9177 heartbeater.cc:347] Connected to a master server at 127.1.188.252:41187
I20260621 07:04:36.415558 9177 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:36.416512 9177 heartbeater.cc:511] Master 127.1.188.252:41187 requested a full tablet report, sending...
I20260621 07:04:36.418526 8713 ts_manager.cc:194] Registered new tserver with Master: 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
Jun 21 07:04:36 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
Jun 21 07:04:36 dist-test-slave-hlq1 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025476, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.253@KRBTEST.COM
I20260621 07:04:36.423084 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.380806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.1.188.252:41187)
0621 07:04:36.394418 (+ 13612us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.394424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.395346 (+ 922us) server_negotiation.cc:407] Connection header received
0621 07:04:36.395437 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.395443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.395528 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.395639 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.400041 (+ 4402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.401478 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.404744 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.405645 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.405756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.410508 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.410515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.410519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.410558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.418977 (+ 8419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.420242 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.420247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.420250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.420500 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.421132 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.421140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.421146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.421514 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.421756 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.422327 (+ 571us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.422599 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13463,"thread_start_us":13359,"threads_started":1}
I20260621 07:04:36.434677 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.356794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:47243 (local address 127.1.188.253:43365)
0621 07:04:36.375246 (+ 18452us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.375254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.406381 (+ 31127us) server_negotiation.cc:407] Connection header received
0621 07:04:36.410700 (+ 4319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.410709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.410812 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.410913 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.413448 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.414750 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.416693 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.417499 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.419016 (+ 1517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.425500 (+ 6484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.425507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.425512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.425557 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.430662 (+ 5105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.432335 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.432341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.432345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.432478 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.432912 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.432919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.432927 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.433328 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.433458 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.434004 (+ 546us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.434243 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18294,"spinlock_wait_cycles":2944,"thread_start_us":37085,"threads_started":1}
I20260621 07:04:36.435626 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.356724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.1.188.195:47243)
0621 07:04:36.406257 (+ 49533us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.406283 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.410469 (+ 4186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.411310 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.411318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.412700 (+ 1382us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:36.412947 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.412961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.414891 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.414897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.416480 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.416499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.419276 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.419283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.419481 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.420082 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:36.420104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:36.425242 (+ 5138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:36.430820 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.430826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.430830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.432164 (+ 1334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.432603 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.432608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.432612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.432773 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.433541 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:36.433547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:36.433719 (+ 172us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.434890 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.435202 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49389,"thread_start_us":56484,"threads_started":1}
I20260621 07:04:36.437789 9179 heartbeater.cc:347] Connected to a master server at 127.1.188.253:43365
I20260621 07:04:36.437963 9179 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:36.438423 9179 heartbeater.cc:511] Master 127.1.188.253:43365 requested a full tablet report, sending...
I20260621 07:04:36.439464 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.344334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:38555)
0621 07:04:36.397635 (+ 53301us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.397659 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.397885 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.400329 (+ 2444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.400336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.401133 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:36.401442 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.401454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.403674 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.403682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.405663 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.405683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.414143 (+ 8460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.414148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.415015 (+ 867us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.417176 (+ 2161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:36.417198 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:36.424981 (+ 7783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:36.432756 (+ 7775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.432761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.432764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.433380 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.434667 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:36.434673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:36.434677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:36.434841 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:36.437151 (+ 2310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:36.437159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:36.437357 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.437514 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.437865 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12262,"thread_start_us":12044,"threads_started":1}
I20260621 07:04:36.439658 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.344306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:38555 (local address 127.1.188.254:37109)
0621 07:04:36.378756 (+ 34450us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.378795 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.397758 (+ 18963us) server_negotiation.cc:407] Connection header received
0621 07:04:36.399852 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.399859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.399972 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.400064 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.401677 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.403442 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.412833 (+ 9391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.413850 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.413989 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.426191 (+ 12202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.426425 (+ 234us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.426432 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.426476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.430870 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.434371 (+ 3501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.434376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.434380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.434506 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.436636 (+ 2130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.436641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.436645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.437000 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.437157 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.439084 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.439310 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34233,"thread_start_us":67767,"threads_started":1}
I20260621 07:04:36.439837 8524 ts_manager.cc:194] Registered new tserver with Master: 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
I20260621 07:04:36.440106 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.358763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.1.188.253:43365)
0621 07:04:36.393854 (+ 35091us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.393861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.393887 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:36.393956 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.393962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.394063 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.398048 (+ 3985us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.398252 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.399601 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.410452 (+ 10851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.411198 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.411324 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.423614 (+ 12290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.423621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.423625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.423665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.430113 (+ 6448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.432733 (+ 2620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.432738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.432742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.433074 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.434666 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.434674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.434677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.435047 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.435256 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.437302 (+ 2046us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.437518 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34958,"spinlock_wait_cycles":2944,"thread_start_us":17961,"threads_started":1}
I20260621 07:04:36.440797 9178 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37109
I20260621 07:04:36.441040 9178 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:36.441802 9178 heartbeater.cc:511] Master 127.1.188.254:37109 requested a full tablet report, sending...
I20260621 07:04:36.442863 7925 ts_manager.cc:194] Re-registered known tserver with Master: 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
I20260621 07:04:36.443485 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.358728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.1.188.254:37109)
0621 07:04:36.412235 (+ 53507us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.412242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.412262 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:04:36.412318 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.412322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.412413 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.412517 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:36.419191 (+ 6674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.420416 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.426265 (+ 5849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.426999 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.427135 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.433709 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:36.433715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:36.433719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:36.433753 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:36.437884 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.438792 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.438799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.438805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.438953 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:36.440680 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:36.440685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:36.440689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:36.441008 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:36.441541 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.442865 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.443070 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52879,"thread_start_us":19587,"threads_started":1}
I20260621 07:04:36.444515 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:38555
I20260621 07:04:36.449083 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:36.458073 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:36.494113 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.479073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.1.188.195:37761)
0621 07:04:36.479943 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.479956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.485707 (+ 5751us) server_negotiation.cc:407] Connection header received
0621 07:04:36.485818 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.485826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.486053 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.486243 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:36.487138 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.488676 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.491355 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.492713 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.492841 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.493101 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.493567 (+ 466us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.493753 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":431,"threads_started":1}
I20260621 07:04:36.494129 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.478764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37761 (local address 127.0.0.1:33910)
0621 07:04:36.484059 (+ 5295us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.484100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.484276 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.486532 (+ 2256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.486537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.486560 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:36.486927 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.486938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.488838 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.488845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.491198 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.491217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.492985 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.492992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.493190 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.493411 (+ 221us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.493511 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.493687 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5116,"mutex_wait_us":173,"thread_start_us":4934,"threads_started":1}
I20260621 07:04:36.494268 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.478216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.1.188.194:43957)
0621 07:04:36.484917 (+ 6701us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.484924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.484948 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:04:36.485030 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.485035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.485250 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.485452 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:36.486075 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.487635 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.491185 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.492684 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.492837 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.493059 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.493801 (+ 742us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.493965 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6394,"thread_start_us":6284,"threads_started":1}
I20260621 07:04:36.494746 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.477997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:43957 (local address 127.0.0.1:34230)
0621 07:04:36.478704 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.478723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.478849 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.485447 (+ 6598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.485452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.485476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:36.485867 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.485879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.487798 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.487804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.490945 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.490964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.492996 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.493001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.493287 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.493411 (+ 124us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.493511 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.493682 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":99,"thread_start_us":400,"threads_started":1}
I20260621 07:04:36.498497 8971 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (DEFAULT_TABLE table=test-table [id=1c59d0c7c94a4bd094c5572215d41607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:36.498558 9112 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (DEFAULT_TABLE table=test-table [id=1c59d0c7c94a4bd094c5572215d41607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:36.499897 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:36.499960 9112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:36.502615 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.484014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38565 (local address 127.0.0.1:59480)
0621 07:04:36.493341 (+ 9327us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.493365 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.494893 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.495694 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.495698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.495726 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:36.496032 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.496044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.497887 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.497892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.499912 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.499927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.501773 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.501780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.501968 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.502002 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.502111 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.502246 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7260,"thread_start_us":7098,"threads_started":1}
I20260621 07:04:36.502656 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.484389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.1.188.193:38565)
0621 07:04:36.488572 (+ 4183us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.488580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.495042 (+ 6462us) server_negotiation.cc:407] Connection header received
0621 07:04:36.495173 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.495181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.495488 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.495690 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:36.496232 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.497726 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.500125 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.501526 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.501689 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.501890 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.502189 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.502355 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4032,"thread_start_us":3926,"threads_started":1}
I20260621 07:04:36.506873 8826 tablet_service.cc:1467] Processing CreateTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (DEFAULT_TABLE table=test-table [id=1c59d0c7c94a4bd094c5572215d41607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:36.508395 8826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:36.512667 9214 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Bootstrap starting.
I20260621 07:04:36.515085 9215 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Bootstrap starting.
I20260621 07:04:36.517299 9214 tablet_bootstrap.cc:654] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:36.518633 9215 tablet_bootstrap.cc:654] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:36.518669 9214 log.cc:824] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:36.519794 9215 log.cc:824] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:36.521679 9214 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: No bootstrap required, opened a new log
I20260621 07:04:36.522003 9214 ts_tablet_manager.cc:1397] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260621 07:04:36.522773 9215 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: No bootstrap required, opened a new log
I20260621 07:04:36.523044 9215 ts_tablet_manager.cc:1397] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 07:04:36.531900 9218 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Bootstrap starting.
I20260621 07:04:36.533188 9214 raft_consensus.cc:348] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.533789 9214 raft_consensus.cc:374] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:36.533900 9214 raft_consensus.cc:729] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf10dfd450b4ed48fe793db0ed11988, State: Initialized, Role: FOLLOWER
I20260621 07:04:36.533746 9215 raft_consensus.cc:348] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.534161 9215 raft_consensus.cc:374] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:36.534281 9215 raft_consensus.cc:729] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d753a88e97545f1958c842e6c28012a, State: Initialized, Role: FOLLOWER
I20260621 07:04:36.534657 9214 consensus_queue.cc:260] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.534821 9215 consensus_queue.cc:260] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.536214 9218 tablet_bootstrap.cc:654] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:36.536755 9214 ts_tablet_manager.cc:1428] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260621 07:04:36.537107 9037 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
I20260621 07:04:36.537799 9218 log.cc:824] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:36.543457 9215 ts_tablet_manager.cc:1428] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260621 07:04:36.543743 9178 heartbeater.cc:503] Master 127.1.188.254:37109 was elected leader, sending a full tablet report...
I20260621 07:04:36.548748 9218 tablet_bootstrap.cc:492] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: No bootstrap required, opened a new log
I20260621 07:04:36.549057 9218 ts_tablet_manager.cc:1397] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20260621 07:04:36.556198 9218 raft_consensus.cc:348] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.556572 9218 raft_consensus.cc:374] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:36.556677 9218 raft_consensus.cc:729] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68f9669d85a4a36bbbc17470fbe03fc, State: Initialized, Role: FOLLOWER
I20260621 07:04:36.557260 9218 consensus_queue.cc:260] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.559167 9218 ts_tablet_manager.cc:1428] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
W20260621 07:04:36.588785 9180 tablet.cc:2369] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:04:36.593672 8894 tablet.cc:2369] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:04:36.708834 9039 tablet.cc:2369] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:36.840399 9222 raft_consensus.cc:482] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:36.840710 9222 raft_consensus.cc:504] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.842348 9222 leader_election.cc:288] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957), 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
I20260621 07:04:36.849973 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.842671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:43957 (local address 127.1.188.193:50413)
0621 07:04:36.843438 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.843463 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.843683 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.844242 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.844247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.844272 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:36.844652 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.844666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.846099 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.846104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.847813 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.847824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.849184 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.849191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.849363 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.849398 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.849503 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.849654 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":305,"threads_started":1}
I20260621 07:04:36.850025 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.842794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:50413 (local address 127.1.188.194:43957)
0621 07:04:36.843025 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.843033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.843552 (+ 519us) server_negotiation.cc:407] Connection header received
0621 07:04:36.843864 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.843870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.843952 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.844019 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:36.844801 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.845956 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.847964 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.848977 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.849108 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.849166 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.849580 (+ 414us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.849765 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:04:36.850526 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.843506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:41077 (local address 127.1.188.195:37761)
0621 07:04:36.843748 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.843758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.844340 (+ 582us) server_negotiation.cc:407] Connection header received
0621 07:04:36.844635 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.844640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.844723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.844790 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:36.845518 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.846480 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.848541 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.849548 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.849692 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.849759 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.850172 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.850330 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 07:04:36.851074 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.843382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37761 (local address 127.1.188.193:41077)
0621 07:04:36.844233 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:36.844247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:36.844411 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:36.844987 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:36.844991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:36.845053 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:36.845361 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.845372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.846651 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.846655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.848382 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:36.848396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.849773 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.849779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:36.849953 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.849988 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:04:36.850712 (+ 724us) client_negotiation.cc:240] Negotiation successful
0621 07:04:36.850841 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":581,"threads_started":1}
I20260621 07:04:36.851025 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9c14f92fc54a85b8e6c31f32d7a21d" candidate_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf10dfd450b4ed48fe793db0ed11988" is_pre_election: true
I20260621 07:04:36.851673 8991 raft_consensus.cc:2454] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68f9669d85a4a36bbbc17470fbe03fc in term 0.
I20260621 07:04:36.852077 9132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9c14f92fc54a85b8e6c31f32d7a21d" candidate_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d753a88e97545f1958c842e6c28012a" is_pre_election: true
I20260621 07:04:36.852419 8781 leader_election.cc:302] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e68f9669d85a4a36bbbc17470fbe03fc, ecf10dfd450b4ed48fe793db0ed11988; no voters:
I20260621 07:04:36.852495 9132 raft_consensus.cc:2454] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68f9669d85a4a36bbbc17470fbe03fc in term 0.
I20260621 07:04:36.852837 9222 raft_consensus.cc:2781] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:36.853012 9222 raft_consensus.cc:482] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:36.853155 9222 raft_consensus.cc:3037] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:36.855340 9222 raft_consensus.cc:504] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.856022 9222 leader_election.cc:288] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [CANDIDATE]: Term 1 election: Requested vote from peers ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957), 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761)
I20260621 07:04:36.856431 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9c14f92fc54a85b8e6c31f32d7a21d" candidate_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf10dfd450b4ed48fe793db0ed11988"
I20260621 07:04:36.856534 9132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d9c14f92fc54a85b8e6c31f32d7a21d" candidate_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d753a88e97545f1958c842e6c28012a"
I20260621 07:04:36.856642 8991 raft_consensus.cc:3037] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:36.856736 9132 raft_consensus.cc:3037] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:36.859926 9132 raft_consensus.cc:2454] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68f9669d85a4a36bbbc17470fbe03fc in term 1.
I20260621 07:04:36.859926 8991 raft_consensus.cc:2454] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68f9669d85a4a36bbbc17470fbe03fc in term 1.
I20260621 07:04:36.860319 8781 leader_election.cc:302] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d753a88e97545f1958c842e6c28012a, e68f9669d85a4a36bbbc17470fbe03fc; no voters:
I20260621 07:04:36.860584 9222 raft_consensus.cc:2781] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:36.861006 9222 raft_consensus.cc:686] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 1 LEADER]: Becoming Leader. State: Replica: e68f9669d85a4a36bbbc17470fbe03fc, State: Running, Role: LEADER
I20260621 07:04:36.861588 9222 consensus_queue.cc:237] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:36.867578 7924 catalog_manager.cc:5495] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc reported cstate change: term changed from 0 to 1, leader changed from <none> to e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193). New cstate: current_term: 1 leader_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260621 07:04:36.945915 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.940493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.1.188.193:38565)
0621 07:04:36.940658 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:36.940662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:36.940679 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:36.940844 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:36.940848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:36.940926 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:36.941047 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:36.941392 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.942253 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.943640 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:36.944208 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:36.944300 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:36.944789 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:36.945364 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:36.945502 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:36.945647 (+ 145us) server_negotiation.cc:299] Negotiation successful
0621 07:04:36.945710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:04:36.954938 8991 raft_consensus.cc:1261] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Refusing update from remote peer e68f9669d85a4a36bbbc17470fbe03fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:36.954954 9132 raft_consensus.cc:1261] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Refusing update from remote peer e68f9669d85a4a36bbbc17470fbe03fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:36.955925 9225 consensus_queue.cc:1035] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:36.956287 9222 consensus_queue.cc:1035] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:36.970654 9228 mvcc.cc:205] Tried to move back new op lower bound from 7299176353597902848 to 7299176353229074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299176353597902848}]
I20260621 07:04:36.980012 9229 mvcc.cc:205] Tried to move back new op lower bound from 7299176353597902848 to 7299176353229074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299176353597902848 or (T in {7299176353597902848})}]
I20260621 07:04:36.982213 9230 mvcc.cc:205] Tried to move back new op lower bound from 7299176353597902848 to 7299176353229074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299176353597902848}]
I20260621 07:04:36.989501 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:36.989801 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:36.996238 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6245152a3fc4468fbbc9bd6194d309aa: Sending DeleteTablet for 3 replicas of tablet 7d9c14f92fc54a85b8e6c31f32d7a21d
I20260621 07:04:36.997846 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:36 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:34230
I20260621 07:04:36.998066 9112 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:36 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:33910
I20260621 07:04:36.998296 8826 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d9c14f92fc54a85b8e6c31f32d7a21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:36 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59480
I20260621 07:04:37.043329 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 7893
I20260621 07:04:37.045405 7955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:37.064965 9238 tablet_replica.cc:330] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: stopping tablet replica
I20260621 07:04:37.065762 9238 raft_consensus.cc:2229] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:37.068204 9238 raft_consensus.cc:2258] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:37.068899 9239 tablet_replica.cc:330] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: stopping tablet replica
I20260621 07:04:37.069492 9239 raft_consensus.cc:2229] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:37.070070 9239 raft_consensus.cc:2258] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:37.073447 9238 ts_tablet_manager.cc:1905] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:37.073447 9239 ts_tablet_manager.cc:1905] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:37.079933 9240 tablet_replica.cc:330] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: stopping tablet replica
I20260621 07:04:37.080384 9238 ts_tablet_manager.cc:1918] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:37.080391 9240 raft_consensus.cc:2229] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:37.080556 9238 log.cc:1196] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/7d9c14f92fc54a85b8e6c31f32d7a21d
I20260621 07:04:37.081022 9240 raft_consensus.cc:2258] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:37.081076 9238 ts_tablet_manager.cc:1939] T 7d9c14f92fc54a85b8e6c31f32d7a21d P ecf10dfd450b4ed48fe793db0ed11988: Deleting consensus metadata
I20260621 07:04:37.083140 9240 ts_tablet_manager.cc:1905] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:37.084383 9239 ts_tablet_manager.cc:1918] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:37.084636 9239 log.cc:1196] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/7d9c14f92fc54a85b8e6c31f32d7a21d
I20260621 07:04:37.085206 9239 ts_tablet_manager.cc:1939] T 7d9c14f92fc54a85b8e6c31f32d7a21d P 5d753a88e97545f1958c842e6c28012a: Deleting consensus metadata
I20260621 07:04:37.094125 9240 ts_tablet_manager.cc:1918] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:37.094317 9240 log.cc:1196] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/7d9c14f92fc54a85b8e6c31f32d7a21d
I20260621 07:04:37.094923 9240 ts_tablet_manager.cc:1939] T 7d9c14f92fc54a85b8e6c31f32d7a21d P e68f9669d85a4a36bbbc17470fbe03fc: Deleting consensus metadata
I20260621 07:04:37.100912 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:36.999743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.1.188.253:43365)
0621 07:04:37.033364 (+ 33621us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:37.033374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:37.033400 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:37.033493 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:37.033502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:37.033621 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:37.033776 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:37.058884 (+ 25108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.060319 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.070754 (+ 10435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.071731 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.071877 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:37.083461 (+ 11584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:37.083469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:37.083474 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:37.083519 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:37.087170 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.088092 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.088098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.088101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.088253 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.088664 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.088669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.088671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.089024 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:37.100117 (+ 11093us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:37.100278 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 07:04:37.100549 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33446,"thread_start_us":41345,"threads_started":1}
I20260621 07:04:37.109434 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.034639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.1.188.252:41187)
0621 07:04:37.067067 (+ 32428us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:37.067076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:37.090284 (+ 23208us) server_negotiation.cc:407] Connection header received
0621 07:04:37.090501 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:37.090507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:37.090598 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:37.090708 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:37.092588 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.093744 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.095843 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.096463 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.096581 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:37.102620 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:37.102627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:37.102632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:37.102685 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:37.106283 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.107123 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.107129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.107132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.107288 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.107737 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.107743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.107746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.108078 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:37.108294 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:37.108839 (+ 545us) server_negotiation.cc:299] Negotiation successful
0621 07:04:37.109068 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32171,"thread_start_us":32440,"threads_started":1}
I20260621 07:04:37.224014 7911 catalog_manager.cc:4841] TS e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565): tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (table test-table [id=1c59d0c7c94a4bd094c5572215d41607]) successfully deleted
I20260621 07:04:37.224363 7911 catalog_manager.cc:4841] TS 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761): tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (table test-table [id=1c59d0c7c94a4bd094c5572215d41607]) successfully deleted
I20260621 07:04:37.224561 7911 catalog_manager.cc:4841] TS ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957): tablet 7d9c14f92fc54a85b8e6c31f32d7a21d (table test-table [id=1c59d0c7c94a4bd094c5572215d41607]) successfully deleted
I20260621 07:04:37.224673 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 7893
I20260621 07:04:37.269371 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.254011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.1.188.252:41187)
0621 07:04:37.254231 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:37.254239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:37.254259 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:04:37.254321 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:37.254326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:37.254402 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:37.254506 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:37.256679 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.257408 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.259110 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.259835 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.259971 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:37.263624 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:37.263632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:37.263639 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:37.263688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:37.266408 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.267348 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.267354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.267357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.267537 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.267917 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.267923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.267926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.268226 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:37.268486 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:37.268841 (+ 355us) server_negotiation.cc:299] Negotiation successful
0621 07:04:37.269055 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 07:04:37.269897 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.251970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.1.188.253:43365)
0621 07:04:37.252212 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:37.252222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:37.253267 (+ 1045us) server_negotiation.cc:407] Connection header received
0621 07:04:37.253602 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:37.253607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:37.253707 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:37.253803 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:37.255832 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.256564 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.258433 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:37.259188 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:37.259330 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:37.262679 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:37.262689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:37.262693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:37.262728 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:37.266394 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.267159 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.267165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.267169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.267347 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:37.267819 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:37.267824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:37.267828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:37.268215 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:37.268464 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:37.269095 (+ 631us) server_negotiation.cc:299] Negotiation successful
0621 07:04:37.269522 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:04:37.981383 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.980369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:38521)
0621 07:04:37.980929 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:37.981114 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":303,"threads_started":1}
W20260621 07:04:37.981978 8892 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:37109 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:37.982985 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.981675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:58747)
0621 07:04:37.982613 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:37.982759 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":754,"threads_started":1}
W20260621 07:04:37.983474 9178 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:37109 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:37.988699 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:37.987909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:44911)
0621 07:04:37.988411 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:37.988536 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
W20260621 07:04:37.989159 9037 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:37109 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.497475 9254 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6245152a3fc4468fbbc9bd6194d309aa)
I20260621 07:04:38.497677 9255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6245152a3fc4468fbbc9bd6194d309aa)
I20260621 07:04:38.497922 9254 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:38.498005 9255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:38.499672 9255 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), d5441c6a0617421f969b9e294261df70 (127.1.188.253:43365)
I20260621 07:04:38.505739 9254 leader_election.cc:288] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252:41187)
I20260621 07:04:38.513505 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.499805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55604)
0621 07:04:38.512881 (+ 13076us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.513172 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12960,"threads_started":1}
W20260621 07:04:38.515388 8699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.515579 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.499807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55590)
0621 07:04:38.514801 (+ 14994us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.515101 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14813,"thread_start_us":15426,"threads_started":1}
W20260621 07:04:38.516407 8510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.524613 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.523797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55628)
0621 07:04:38.524076 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.524300 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260621 07:04:38.524720 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.524317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55634)
0621 07:04:38.524455 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.524545 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 07:04:38.525049 8510 leader_election.cc:334] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
W20260621 07:04:38.525135 8699 leader_election.cc:334] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.525650 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.500449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43365 (local address 127.0.0.1:48770)
0621 07:04:38.513891 (+ 13442us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.513912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:38.514062 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:38.516795 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:38.516802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:38.516840 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:38.517213 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:38.517230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.519612 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.519622 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:38.522659 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:38.522680 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.524760 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.524766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:38.524966 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:38.525055 (+ 89us) client_negotiation.cc:769] Sending connection context
0621 07:04:38.525154 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:04:38.525320 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13344,"mutex_wait_us":80,"thread_start_us":11999,"threads_started":1}
I20260621 07:04:38.526260 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.499851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:41187 (local address 127.0.0.1:57692)
0621 07:04:38.515716 (+ 15865us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.515735 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:38.515891 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:38.516829 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:38.516839 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:38.516881 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:38.517417 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:38.517437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.519956 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.519967 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:38.523055 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:38.523080 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.525396 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.525402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:38.525616 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:38.525646 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:04:38.525780 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 07:04:38.525928 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15718,"thread_start_us":14534,"threads_started":1}
I20260621 07:04:38.526459 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.504953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.1.188.252:41187)
0621 07:04:38.512660 (+ 7707us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:38.512668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:38.515858 (+ 3190us) server_negotiation.cc:407] Connection header received
0621 07:04:38.516274 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:38.516283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:38.516418 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:38.516552 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:38.517963 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.519714 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.523342 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.524888 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.525052 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:38.525202 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:38.525955 (+ 753us) server_negotiation.cc:299] Negotiation successful
0621 07:04:38.526096 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":7544,"thread_start_us":7705,"threads_started":1}
I20260621 07:04:38.526593 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.515601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.1.188.253:43365)
0621 07:04:38.516436 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:38.516443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:38.516461 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:04:38.516534 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:38.516540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:38.516620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:38.517397 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:38.517710 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.519366 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.522992 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:38.524575 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:38.525869 (+ 1294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:38.525993 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:38.526037 (+ 44us) server_negotiation.cc:299] Negotiation successful
0621 07:04:38.526169 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":483,"threads_started":1}
I20260621 07:04:38.527217 8534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5441c6a0617421f969b9e294261df70" is_pre_election: true
I20260621 07:04:38.527217 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5441c6a0617421f969b9e294261df70" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" is_pre_election: true
I20260621 07:04:38.527681 8534 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5acc76e6da4ec9bc00c7f1f629c578 in term 1.
I20260621 07:04:38.527681 8723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5441c6a0617421f969b9e294261df70 in term 1.
I20260621 07:04:38.528275 8698 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d5acc76e6da4ec9bc00c7f1f629c578, d5441c6a0617421f969b9e294261df70; no voters: 6245152a3fc4468fbbc9bd6194d309aa
I20260621 07:04:38.528347 8509 leader_election.cc:302] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d5acc76e6da4ec9bc00c7f1f629c578, d5441c6a0617421f969b9e294261df70; no voters: 6245152a3fc4468fbbc9bd6194d309aa
I20260621 07:04:38.528770 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:04:38.528824 9254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 07:04:38.528923 9255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6245152a3fc4468fbbc9bd6194d309aa)
I20260621 07:04:38.528959 9254 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6245152a3fc4468fbbc9bd6194d309aa)
I20260621 07:04:38.529037 9255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:04:38.529062 9254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:04:38.531533 9254 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:38.531625 9255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:38.532358 9254 leader_election.cc:288] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 election: Requested vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252:41187)
I20260621 07:04:38.532572 9255 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 election: Requested vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), d5441c6a0617421f969b9e294261df70 (127.1.188.253:43365)
I20260621 07:04:38.533066 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5441c6a0617421f969b9e294261df70" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
I20260621 07:04:38.533185 8534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d5441c6a0617421f969b9e294261df70"
I20260621 07:04:38.533334 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.532791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55644)
0621 07:04:38.532986 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.533126 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 07:04:38.533419 8723 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d5441c6a0617421f969b9e294261df70 in current term 2: Already voted for candidate 9d5acc76e6da4ec9bc00c7f1f629c578 in this term.
I20260621 07:04:38.533496 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.532561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55638)
0621 07:04:38.533104 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.533210 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
I20260621 07:04:38.533533 8534 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d5acc76e6da4ec9bc00c7f1f629c578 in current term 2: Already voted for candidate d5441c6a0617421f969b9e294261df70 in this term.
I20260621 07:04:38.534654 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.534208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55660)
0621 07:04:38.534390 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.534497 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 07:04:38.534847 8699 leader_election.cc:334] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.534962 8699 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d5acc76e6da4ec9bc00c7f1f629c578; no voters: 6245152a3fc4468fbbc9bd6194d309aa, d5441c6a0617421f969b9e294261df70
I20260621 07:04:38.535221 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.534764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55676)
0621 07:04:38.534893 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.534998 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 07:04:38.535298 9255 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260621 07:04:38.535477 8510 leader_election.cc:334] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:38.535619 8510 leader_election.cc:302] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5441c6a0617421f969b9e294261df70; no voters: 6245152a3fc4468fbbc9bd6194d309aa, 9d5acc76e6da4ec9bc00c7f1f629c578
I20260621 07:04:38.535902 9254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260621 07:04:38.983762 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.982740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:48763)
0621 07:04:38.983314 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.983477 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":321,"threads_started":1}
I20260621 07:04:38.985589 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.984727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:53581)
0621 07:04:38.985252 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.985373 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260621 07:04:38.992076 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:38.989805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:53163)
0621 07:04:38.991791 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:38.991883 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1886,"thread_start_us":1779,"threads_started":1}
I20260621 07:04:39.986078 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:39.984984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:41335)
0621 07:04:39.985628 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:39.985753 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":376,"threads_started":1}
I20260621 07:04:39.987543 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:39.986478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:35791)
0621 07:04:39.987093 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:39.987246 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":374,"threads_started":1}
I20260621 07:04:39.993614 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:39.992879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:37189)
0621 07:04:39.993344 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:39.993437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260621 07:04:40.110482 9270 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:40.110780 9270 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:40.111713 9270 leader_election.cc:288] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252:41187)
I20260621 07:04:40.112465 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5441c6a0617421f969b9e294261df70" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" is_pre_election: true
I20260621 07:04:40.112790 8723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5441c6a0617421f969b9e294261df70 in term 2.
I20260621 07:04:40.113054 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.111977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55898)
0621 07:04:40.112636 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.112793 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":370,"threads_started":1}
I20260621 07:04:40.113366 8509 leader_election.cc:302] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d5acc76e6da4ec9bc00c7f1f629c578, d5441c6a0617421f969b9e294261df70; no voters:
I20260621 07:04:40.113782 9270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260621 07:04:40.113904 9270 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:40.114017 9270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:04:40.114748 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.114309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55906)
0621 07:04:40.114507 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.114601 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 07:04:40.115044 8510 leader_election.cc:334] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:40.116978 9270 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:40.117682 9270 leader_election.cc:288] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 election: Requested vote from peers 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109), 9d5acc76e6da4ec9bc00c7f1f629c578 (127.1.188.252:41187)
I20260621 07:04:40.118391 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.117970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55912)
0621 07:04:40.118144 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.118246 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 07:04:40.118345 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5441c6a0617421f969b9e294261df70" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578"
I20260621 07:04:40.118695 8723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:04:40.119293 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.118979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55914)
0621 07:04:40.119124 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.119196 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:04:40.119472 8510 leader_election.cc:334] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
I20260621 07:04:40.121873 8723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5441c6a0617421f969b9e294261df70 in term 3.
I20260621 07:04:40.122434 8509 leader_election.cc:302] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d5acc76e6da4ec9bc00c7f1f629c578, d5441c6a0617421f969b9e294261df70; no voters: 6245152a3fc4468fbbc9bd6194d309aa
I20260621 07:04:40.122880 9270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:04:40.123370 9270 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [term 3 LEADER]: Becoming Leader. State: Replica: d5441c6a0617421f969b9e294261df70, State: Running, Role: LEADER
I20260621 07:04:40.123935 9270 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } }
I20260621 07:04:40.126156 9273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5441c6a0617421f969b9e294261df70. Latest consensus state: current_term: 3 leader_uuid: "d5441c6a0617421f969b9e294261df70" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:40.126678 9273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:40.127563 9275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:40.131989 9275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:40.133154 9275 catalog_manager.cc:1259] Loaded cluster ID: 091540ad3b1442a7bc021c36f7783afe
I20260621 07:04:40.133257 9275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:40.134686 9275 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:40.135640 9275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Loaded TSK: 0
I20260621 07:04:40.136705 9275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:04:40.179231 8523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:40.187029 8523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:40.191843 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.191349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55924)
0621 07:04:40.191578 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.191669 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260621 07:04:40.192076 8723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [term 3 FOLLOWER]: Refusing update from remote peer d5441c6a0617421f969b9e294261df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260621 07:04:40.193039 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.192544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55926)
0621 07:04:40.192716 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.192837 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 07:04:40.192854 9273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260621 07:04:40.193251 8510 consensus_peers.cc:487] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 -> Peer 6245152a3fc4468fbbc9bd6194d309aa (127.1.188.254:37109): Couldn't send request to peer 6245152a3fc4468fbbc9bd6194d309aa. Status: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 07:04:40.196094 9277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5441c6a0617421f969b9e294261df70. Latest consensus state: current_term: 3 leader_uuid: "d5441c6a0617421f969b9e294261df70" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:40.196440 9277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:40.197208 9276 mvcc.cc:205] Tried to move back new op lower bound from 7299176366858616832 to 7299176366591713280. Current Snapshot: MvccSnapshot[applied={T|T < 7299176366858616832}]
I20260621 07:04:40.198007 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d5441c6a0617421f969b9e294261df70" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:40.198387 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70 [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:40.198302 9277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d5441c6a0617421f969b9e294261df70" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6245152a3fc4468fbbc9bd6194d309aa" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37109 } } peers { permanent_uuid: "d5441c6a0617421f969b9e294261df70" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43365 } attrs { promote: false } } peers { permanent_uuid: "9d5acc76e6da4ec9bc00c7f1f629c578" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 41187 } attrs { promote: false } } }
I20260621 07:04:40.198606 9277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5acc76e6da4ec9bc00c7f1f629c578 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:04:40.226418 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.216521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.1.188.194:43957)
0621 07:04:40.217210 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.217220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.217247 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:04:40.217356 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.217365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.217480 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.217645 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:40.218820 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.220419 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.224357 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.225366 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.225526 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.225655 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.225904 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.226051 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":552,"threads_started":1}
I20260621 07:04:40.226478 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.216030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:43957 (local address 127.0.0.1:34246)
0621 07:04:40.216698 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.216717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:40.216836 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:40.217995 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:40.218004 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:40.218042 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:40.218518 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.218535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.220672 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.220680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.224158 (+ 3478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.224179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.225550 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.225556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.225732 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.225757 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:04:40.225863 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:04:40.226004 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":28,"thread_start_us":346,"threads_started":1}
I20260621 07:04:40.228215 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.217459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38565 (local address 127.0.0.1:59500)
0621 07:04:40.217988 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.218011 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:40.218125 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:40.220333 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:40.220338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:40.220366 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:40.221130 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.221139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.222598 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.222604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.224602 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.224616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.227081 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.227090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.227360 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.227499 (+ 139us) client_negotiation.cc:769] Sending connection context
0621 07:04:40.227601 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 07:04:40.227772 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":4,"thread_start_us":858,"threads_started":1}
I20260621 07:04:40.228453 8971 tablet_service.cc:1467] Processing CreateTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 (DEFAULT_TABLE table=test-table [id=9be6ded012fd4de4a7eb6c7830265f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:40.228997 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b48f5d5e944f1eb76612e0efac5647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:40.229408 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.217981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.1.188.193:38565)
0621 07:04:40.220027 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.220032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.220049 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:40.220115 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.220119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.220182 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.220304 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:40.221307 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.222420 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.224774 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.226567 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.226748 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.226911 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.228665 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.228908 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1934,"thread_start_us":1800,"threads_started":1}
I20260621 07:04:40.230283 8826 tablet_service.cc:1467] Processing CreateTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 (DEFAULT_TABLE table=test-table [id=9be6ded012fd4de4a7eb6c7830265f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:40.230752 8826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b48f5d5e944f1eb76612e0efac5647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:40.236498 9287 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Bootstrap starting.
I20260621 07:04:40.238138 9288 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Bootstrap starting.
I20260621 07:04:40.238617 9287 tablet_bootstrap.cc:654] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:40.238953 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.215708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37761 (local address 127.0.0.1:33914)
0621 07:04:40.215895 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.215918 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:40.216059 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:40.232581 (+ 16522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:40.232586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:40.232619 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:40.232929 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.232981 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.234648 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.234655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.236202 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.236220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.238113 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.238119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.238286 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.238321 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:04:40.238416 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:04:40.238611 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 07:04:40.239753 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.215829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.1.188.195:37761)
0621 07:04:40.232222 (+ 16393us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.232226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.232246 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:04:40.232321 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.232325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.232395 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.232537 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:40.233327 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.234508 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.236408 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.237705 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.237854 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.237960 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.239170 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.239351 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16083,"thread_start_us":16186,"threads_started":1}
I20260621 07:04:40.240072 9288 tablet_bootstrap.cc:654] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:40.240623 9112 tablet_service.cc:1467] Processing CreateTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 (DEFAULT_TABLE table=test-table [id=9be6ded012fd4de4a7eb6c7830265f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:40.241127 9112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b48f5d5e944f1eb76612e0efac5647. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:40.241504 9287 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: No bootstrap required, opened a new log
I20260621 07:04:40.241657 9287 ts_tablet_manager.cc:1397] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 07:04:40.242534 9287 raft_consensus.cc:348] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.242755 9288 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: No bootstrap required, opened a new log
I20260621 07:04:40.242823 9287 raft_consensus.cc:374] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:40.242909 9288 ts_tablet_manager.cc:1397] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 07:04:40.242949 9287 raft_consensus.cc:729] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf10dfd450b4ed48fe793db0ed11988, State: Initialized, Role: FOLLOWER
I20260621 07:04:40.243273 9287 consensus_queue.cc:260] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.243844 9288 raft_consensus.cc:348] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.244163 9288 raft_consensus.cc:374] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:40.244297 9287 ts_tablet_manager.cc:1428] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:40.244313 9288 raft_consensus.cc:729] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68f9669d85a4a36bbbc17470fbe03fc, State: Initialized, Role: FOLLOWER
I20260621 07:04:40.244752 9288 consensus_queue.cc:260] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.245887 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.245012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:46481)
0621 07:04:40.245367 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.245654 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260621 07:04:40.247511 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.246814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:59213)
0621 07:04:40.247142 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.247269 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260621 07:04:40.248728 9293 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Bootstrap starting.
I20260621 07:04:40.250881 9288 ts_tablet_manager.cc:1428] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260621 07:04:40.252207 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.251765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:52153)
0621 07:04:40.251952 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.252053 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 07:04:40.252475 9293 tablet_bootstrap.cc:654] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:40.252493 9038 heartbeater.cc:503] Master 127.1.188.253:43365 was elected leader, sending a full tablet report...
I20260621 07:04:40.254324 8893 heartbeater.cc:503] Master 127.1.188.253:43365 was elected leader, sending a full tablet report...
I20260621 07:04:40.254906 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.254398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:51839)
0621 07:04:40.254630 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.254719 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260621 07:04:40.255529 9293 tablet_bootstrap.cc:492] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: No bootstrap required, opened a new log
I20260621 07:04:40.255694 9293 ts_tablet_manager.cc:1397] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260621 07:04:40.256529 9293 raft_consensus.cc:348] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.256729 9293 raft_consensus.cc:374] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:40.256804 9293 raft_consensus.cc:729] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d753a88e97545f1958c842e6c28012a, State: Initialized, Role: FOLLOWER
I20260621 07:04:40.257010 9293 consensus_queue.cc:260] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.257776 9293 ts_tablet_manager.cc:1428] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:04:40.258675 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.258202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:45283)
0621 07:04:40.258404 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.258517 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 07:04:40.259812 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.259444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:46605)
0621 07:04:40.259583 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.259682 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260621 07:04:40.260353 9179 heartbeater.cc:503] Master 127.1.188.253:43365 was elected leader, sending a full tablet report...
I20260621 07:04:40.326507 9291 raft_consensus.cc:482] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:40.326857 9291 raft_consensus.cc:504] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.328892 9291 leader_election.cc:288] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761), e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:40.336938 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.329238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37761 (local address 127.1.188.194:45613)
0621 07:04:40.329447 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.329484 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:40.329642 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:40.330263 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:40.330271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:40.330326 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:40.330608 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.330617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.331940 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.331946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.334119 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.334132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.336080 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.336088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.336342 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.336382 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:04:40.336481 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:04:40.336634 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 07:04:40.336948 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.329447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:45613 (local address 127.1.188.195:37761)
0621 07:04:40.329684 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.329692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.329718 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:40.329816 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.329822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.329910 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.330024 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:40.330781 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.331801 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.334320 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.335742 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.335912 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.335995 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.336538 (+ 543us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.336686 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:04:40.337005 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.329873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:48113 (local address 127.1.188.193:38565)
0621 07:04:40.330166 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.330172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.330537 (+ 365us) server_negotiation.cc:407] Connection header received
0621 07:04:40.330757 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.330764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.330840 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.330943 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:40.331610 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.332488 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.334735 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.335993 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.336190 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.336267 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.336608 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.336742 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 07:04:40.337705 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.329671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38565 (local address 127.1.188.194:48113)
0621 07:04:40.330425 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.330441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:40.330611 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:40.330954 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:40.330958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:40.330980 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:40.331390 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.331449 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.332627 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.332635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.334559 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:40.334573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.336212 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.336219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:40.336398 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.336421 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:04:40.337229 (+ 808us) client_negotiation.cc:240] Negotiation successful
0621 07:04:40.337343 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":377,"threads_started":1}
I20260621 07:04:40.338146 9132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b48f5d5e944f1eb76612e0efac5647" candidate_uuid: "ecf10dfd450b4ed48fe793db0ed11988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d753a88e97545f1958c842e6c28012a" is_pre_election: true
I20260621 07:04:40.338461 9132 raft_consensus.cc:2454] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf10dfd450b4ed48fe793db0ed11988 in term 0.
I20260621 07:04:40.339071 8926 leader_election.cc:302] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d753a88e97545f1958c842e6c28012a, ecf10dfd450b4ed48fe793db0ed11988; no voters:
I20260621 07:04:40.339124 8846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b48f5d5e944f1eb76612e0efac5647" candidate_uuid: "ecf10dfd450b4ed48fe793db0ed11988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" is_pre_election: true
I20260621 07:04:40.339593 9291 raft_consensus.cc:2781] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:40.339643 8846 raft_consensus.cc:2454] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf10dfd450b4ed48fe793db0ed11988 in term 0.
I20260621 07:04:40.339726 9291 raft_consensus.cc:482] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:40.339812 9291 raft_consensus.cc:3037] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:40.342226 9291 raft_consensus.cc:504] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.342916 9291 leader_election.cc:288] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [CANDIDATE]: Term 1 election: Requested vote from peers 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761), e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565)
I20260621 07:04:40.343261 9132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b48f5d5e944f1eb76612e0efac5647" candidate_uuid: "ecf10dfd450b4ed48fe793db0ed11988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d753a88e97545f1958c842e6c28012a"
I20260621 07:04:40.343506 9132 raft_consensus.cc:3037] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:40.343465 8846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b48f5d5e944f1eb76612e0efac5647" candidate_uuid: "ecf10dfd450b4ed48fe793db0ed11988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e68f9669d85a4a36bbbc17470fbe03fc"
I20260621 07:04:40.343716 8846 raft_consensus.cc:3037] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:40.346352 8846 raft_consensus.cc:2454] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf10dfd450b4ed48fe793db0ed11988 in term 1.
I20260621 07:04:40.346635 9132 raft_consensus.cc:2454] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf10dfd450b4ed48fe793db0ed11988 in term 1.
I20260621 07:04:40.346756 8926 leader_election.cc:302] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e68f9669d85a4a36bbbc17470fbe03fc, ecf10dfd450b4ed48fe793db0ed11988; no voters:
I20260621 07:04:40.347129 9291 raft_consensus.cc:2781] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:40.347682 9291 raft_consensus.cc:686] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 1 LEADER]: Becoming Leader. State: Replica: ecf10dfd450b4ed48fe793db0ed11988, State: Running, Role: LEADER
I20260621 07:04:40.348188 9291 consensus_queue.cc:237] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } }
I20260621 07:04:40.348733 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.348254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:55433)
0621 07:04:40.348445 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.348605 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:04:40.352470 8523 catalog_manager.cc:5495] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194). New cstate: current_term: 1 leader_uuid: "ecf10dfd450b4ed48fe793db0ed11988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecf10dfd450b4ed48fe793db0ed11988" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 } health_report { overall_health: UNKNOWN } } }
I20260621 07:04:40.377825 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.372145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.1.188.194:43957)
0621 07:04:40.372421 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.372430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.372449 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:04:40.372559 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.372562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.372661 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.372832 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:40.373224 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.373976 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.375384 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.375866 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.376011 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.376473 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:40.377145 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:40.377318 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.377426 (+ 108us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.377520 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260621 07:04:40.382912 9132 raft_consensus.cc:1261] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Refusing update from remote peer ecf10dfd450b4ed48fe793db0ed11988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:40.383260 8846 raft_consensus.cc:1261] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Refusing update from remote peer ecf10dfd450b4ed48fe793db0ed11988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:40.383843 9297 consensus_queue.cc:1035] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d753a88e97545f1958c842e6c28012a" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:40.384334 9291 consensus_queue.cc:1035] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e68f9669d85a4a36bbbc17470fbe03fc" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:40.387688 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.387240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:43045)
0621 07:04:40.387383 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.387527 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260621 07:04:40.387702 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.387041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:40641)
0621 07:04:40.387295 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.387444 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260621 07:04:40.394583 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.393610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.193:34205)
0621 07:04:40.394159 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.394287 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
I20260621 07:04:40.401772 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.399234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:37025)
0621 07:04:40.401419 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.401545 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2056}
I20260621 07:04:40.403059 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.402519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.194:60323)
0621 07:04:40.402744 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.402845 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260621 07:04:40.410061 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.409212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.1.188.195:42399)
0621 07:04:40.409607 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.409771 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260621 07:04:40.421599 8524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:40.421948 8524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:40.431646 8524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5441c6a0617421f969b9e294261df70: Sending DeleteTablet for 3 replicas of tablet 01b48f5d5e944f1eb76612e0efac5647
I20260621 07:04:40.433177 8826 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:40 UTC) from {username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:59500
I20260621 07:04:40.433669 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:40 UTC) from {username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:34246
I20260621 07:04:40.433722 9112 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b48f5d5e944f1eb76612e0efac5647 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:40 UTC) from {username='oryx', principal='oryx/127.1.188.253@KRBTEST.COM'} at 127.0.0.1:33914
I20260621 07:04:40.434607 9312 tablet_replica.cc:330] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: stopping tablet replica
I20260621 07:04:40.434937 9312 raft_consensus.cc:2229] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:40.435890 9312 raft_consensus.cc:2258] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:40.437304 9312 ts_tablet_manager.cc:1905] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:40.441846 9313 tablet_replica.cc:330] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: stopping tablet replica
I20260621 07:04:40.442117 9313 raft_consensus.cc:2229] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:40.442521 9313 raft_consensus.cc:2258] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:40.443943 9313 ts_tablet_manager.cc:1905] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:40.444007 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8693
I20260621 07:04:40.446283 9312 ts_tablet_manager.cc:1918] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:40.446558 9312 log.cc:1196] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/01b48f5d5e944f1eb76612e0efac5647
I20260621 07:04:40.447137 9312 ts_tablet_manager.cc:1939] T 01b48f5d5e944f1eb76612e0efac5647 P ecf10dfd450b4ed48fe793db0ed11988: Deleting consensus metadata
I20260621 07:04:40.448868 8510 catalog_manager.cc:4841] TS ecf10dfd450b4ed48fe793db0ed11988 (127.1.188.194:43957): tablet 01b48f5d5e944f1eb76612e0efac5647 (table test-table [id=9be6ded012fd4de4a7eb6c7830265f47]) successfully deleted
I20260621 07:04:40.452721 9311 tablet_replica.cc:330] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: stopping tablet replica
I20260621 07:04:40.453099 9311 raft_consensus.cc:2229] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:40.453542 9311 raft_consensus.cc:2258] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:40.455765 9311 ts_tablet_manager.cc:1905] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:40.458402 9313 ts_tablet_manager.cc:1918] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:40.458627 9313 log.cc:1196] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/01b48f5d5e944f1eb76612e0efac5647
I20260621 07:04:40.459276 9313 ts_tablet_manager.cc:1939] T 01b48f5d5e944f1eb76612e0efac5647 P 5d753a88e97545f1958c842e6c28012a: Deleting consensus metadata
I20260621 07:04:40.461083 8510 catalog_manager.cc:4841] TS 5d753a88e97545f1958c842e6c28012a (127.1.188.195:37761): tablet 01b48f5d5e944f1eb76612e0efac5647 (table test-table [id=9be6ded012fd4de4a7eb6c7830265f47]) successfully deleted
I20260621 07:04:40.465560 9311 ts_tablet_manager.cc:1918] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:40.465895 9311 log.cc:1196] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/01b48f5d5e944f1eb76612e0efac5647
I20260621 07:04:40.466472 9311 ts_tablet_manager.cc:1939] T 01b48f5d5e944f1eb76612e0efac5647 P e68f9669d85a4a36bbbc17470fbe03fc: Deleting consensus metadata
I20260621 07:04:40.468988 8510 catalog_manager.cc:4841] TS e68f9669d85a4a36bbbc17470fbe03fc (127.1.188.193:38565): tablet 01b48f5d5e944f1eb76612e0efac5647 (table test-table [id=9be6ded012fd4de4a7eb6c7830265f47]) successfully deleted
I20260621 07:04:40.482393 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.445050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.1.188.193:38565)
0621 07:04:40.445320 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.445327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.454140 (+ 8813us) server_negotiation.cc:407] Connection header received
0621 07:04:40.455994 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.456001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.456103 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.456186 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:40.463593 (+ 7407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.465047 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.467531 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.468414 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.468547 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.473223 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:40.473252 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:40.473282 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:40.473355 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:40.478279 (+ 4924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.479269 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.479283 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.479289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.479550 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.480044 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.480052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.480058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.480740 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:40.481210 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.481459 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.481792 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260621 07:04:40.483289 8886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:40.673411 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.672769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55934)
0621 07:04:40.673045 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.673186 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260621 07:04:40.674960 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.674488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37109 (local address 127.0.0.1:55950)
0621 07:04:40.674701 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:40.674785 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260621 07:04:40.717546 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8693
I20260621 07:04:40.755489 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8903
I20260621 07:04:40.776160 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.756452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.1.188.194:43957)
0621 07:04:40.756704 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.756712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.756867 (+ 155us) server_negotiation.cc:407] Connection header received
0621 07:04:40.757103 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.757109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.757216 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.757357 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:40.759511 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.760756 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.763078 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.763776 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.763918 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.767992 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:40.768009 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:40.768032 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:40.768073 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:40.772483 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.773462 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.773474 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.773478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.773704 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.774190 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.774195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.774199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.774704 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:40.775025 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.775447 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.775704 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 07:04:40.777204 9031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:40.907364 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8903
I20260621 07:04:40.929023 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 9049
I20260621 07:04:40.943624 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:40.929991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.1.188.195:37761)
0621 07:04:40.930485 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:40.930490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:40.930508 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:04:40.930575 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:40.930579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:40.930655 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:40.930758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:40.932436 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.933203 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.934483 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:40.934944 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:40.935059 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:40.937728 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:40.937744 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:40.937763 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:40.937808 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:40.940812 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.941522 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.941535 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.941538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.941679 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:40.942063 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:40.942066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:40.942068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:40.942428 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:40.942641 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:40.942978 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 07:04:40.943193 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260621 07:04:40.944304 9172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:41.039943 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 9049
I20260621 07:04:41.057462 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8387
I20260621 07:04:41.058554 8554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:41.181967 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8387
I20260621 07:04:41.203789 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 8486
I20260621 07:04:41.205008 8743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:41.320639 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 8486
2026-06-21T07:04:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12007 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:41 dist-test-slave-hlq1 krb5kdc[9323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:41 dist-test-slave-hlq1 krb5kdc[9323](info): set up 2 sockets
Jun 21 07:04:41 dist-test-slave-hlq1 krb5kdc[9323](info): commencing operation
krb5kdc: starting...
W20260621 07:04:42.429039 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:42 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab.
Entry for principal oryx/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab.
I20260621 07:04:42.676978 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33173
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab
--principal=oryx/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:33173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:43.003202 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:43.003636 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:43.003809 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:43.011128 9336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:43.011304 9336 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 07:04:43.011389 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:43.011536 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:43.011618 9336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:43.011711 9336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:43.022154 9336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:33173
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.254.keytab
--principal=oryx/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:43.023608 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:43.024963 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:43.037185 9342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:43.037142 9344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:43.037420 9341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:43.037745 9336 server_base.cc:1034] running on GCE node
I20260621 07:04:43.038596 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:43.040045 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:43.041316 9336 hybrid_clock.cc:648] HybridClock initialized: now 1782025483041231 us; error 64 us; skew 500 ppm
I20260621 07:04:43.041695 9336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:43 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:43.044817 9336 init.cc:377] Logged in from keytab as oryx/127.1.188.254@KRBTEST.COM (short username oryx)
I20260621 07:04:43.046239 9336 webserver.cc:460] Webserver started at http://127.1.188.254:33461/ using document root <none> and password file <none>
I20260621 07:04:43.046929 9336 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:43.047052 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:43.047374 9336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:43.050313 9336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "deda2fd32dbb414ca4956ca7ac38c46c"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "91d2c77435e13d6dac6067db1298a6a2"
server_key_iv: "ec80891ffdf5e0c0c7992e857b6b078a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.051118 9336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "deda2fd32dbb414ca4956ca7ac38c46c"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "91d2c77435e13d6dac6067db1298a6a2"
server_key_iv: "ec80891ffdf5e0c0c7992e857b6b078a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.056265 9336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:04:43.060002 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:43.061522 9336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:04:43.061714 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "deda2fd32dbb414ca4956ca7ac38c46c"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "91d2c77435e13d6dac6067db1298a6a2"
server_key_iv: "ec80891ffdf5e0c0c7992e857b6b078a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.061933 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:43.087512 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:43.091416 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:43.091710 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:43.113286 9336 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:33173
I20260621 07:04:43.113319 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:33173 every 8 connection(s)
I20260621 07:04:43.114666 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:43.117966 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 9336
I20260621 07:04:43.118176 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:43.118788 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:43.119037 1779 external_mini_cluster.cc:1428] Setting key bbf8ed5e1fcb1747864a4df138b28c88
I20260621 07:04:43.133612 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Bootstrap starting.
Jun 21 07:04:43 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:43.137543 9413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:43.139005 9413 log.cc:824] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:43.142675 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: No bootstrap required, opened a new log
I20260621 07:04:43.144556 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:43.121400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.1.188.254:33173)
0621 07:04:43.122243 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:43.122257 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:43.122296 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:04:43.123919 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:43.123940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:43.124319 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:43.124640 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:43.126460 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.128043 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.129836 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.130599 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.130745 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:43.135687 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:43.135699 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:43.135722 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:43.135777 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:43.139915 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.140838 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.140848 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.140861 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.141047 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.141472 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.141477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.141480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.142165 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:43.142439 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:43.142813 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 07:04:43.143259 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":426,"threads_started":1}
I20260621 07:04:43.151052 9413 raft_consensus.cc:348] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } }
I20260621 07:04:43.151470 9413 raft_consensus.cc:374] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:43.151564 9413 raft_consensus.cc:729] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deda2fd32dbb414ca4956ca7ac38c46c, State: Initialized, Role: FOLLOWER
I20260621 07:04:43.152100 9413 consensus_queue.cc:260] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } }
I20260621 07:04:43.152310 9413 raft_consensus.cc:388] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:43.152408 9413 raft_consensus.cc:482] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:43.152531 9413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:43.154951 9413 raft_consensus.cc:504] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } }
I20260621 07:04:43.155452 9413 leader_election.cc:302] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deda2fd32dbb414ca4956ca7ac38c46c; no voters:
I20260621 07:04:43.156008 9413 leader_election.cc:288] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:43.156176 9418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:43.156893 9418 raft_consensus.cc:686] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [term 1 LEADER]: Becoming Leader. State: Replica: deda2fd32dbb414ca4956ca7ac38c46c, State: Running, Role: LEADER
I20260621 07:04:43.157471 9418 consensus_queue.cc:237] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } }
I20260621 07:04:43.158152 9413 sys_catalog.cc:564] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:43.160501 9420 sys_catalog.cc:455] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader deda2fd32dbb414ca4956ca7ac38c46c. Latest consensus state: current_term: 1 leader_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } } }
I20260621 07:04:43.160526 9419 sys_catalog.cc:455] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deda2fd32dbb414ca4956ca7ac38c46c" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33173 } } }
I20260621 07:04:43.161126 9420 sys_catalog.cc:458] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:43.161247 9419 sys_catalog.cc:458] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:43.162124 9427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:43.167629 9427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:43.178232 9427 catalog_manager.cc:1347] Generated new cluster ID: 7496d81ef5ec4237975e5237475146f4
I20260621 07:04:43.178385 9427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:43.195295 9427 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:43.196635 9427 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:43.206049 9427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Generated new TSK 0
I20260621 07:04:43.206889 9427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
I20260621 07:04:43.270195 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33173
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:43.622660 9439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:43.623061 9439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:43.623253 9439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:43.633260 9439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:43.633370 9439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:43.633455 9439 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:04:43.633536 9439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:43.644546 9439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:43.646183 9439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:43.647913 9439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:43.662604 9445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:43.662671 9444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:43.662696 9447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:43.663885 9439 server_base.cc:1034] running on GCE node
I20260621 07:04:43.664577 9439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:43.665696 9439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:43.666976 9439 hybrid_clock.cc:648] HybridClock initialized: now 1782025483666877 us; error 71 us; skew 500 ppm
I20260621 07:04:43.667357 9439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:43 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:43.671002 9439 init.cc:377] Logged in from keytab as oryx/127.1.188.193@KRBTEST.COM (short username oryx)
I20260621 07:04:43.672873 9439 webserver.cc:460] Webserver started at http://127.1.188.193:35127/ using document root <none> and password file <none>
I20260621 07:04:43.673712 9439 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:43.673846 9439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:43.674220 9439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:43.677853 9439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "aaf37f2ba46d43828fd871dd6f7e388a"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "73c72d8f6ca937871831edd8ba2ef1af"
server_key_iv: "9c7b3c358cad5d5181c4cc07dd7e441a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.678741 9439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "aaf37f2ba46d43828fd871dd6f7e388a"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "73c72d8f6ca937871831edd8ba2ef1af"
server_key_iv: "9c7b3c358cad5d5181c4cc07dd7e441a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.684684 9439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 07:04:43.688787 9454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:43.690364 9439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:43.690569 9439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "aaf37f2ba46d43828fd871dd6f7e388a"
format_stamp: "Formatted at 2026-06-21 07:04:43 on dist-test-slave-hlq1"
server_key: "73c72d8f6ca937871831edd8ba2ef1af"
server_key_iv: "9c7b3c358cad5d5181c4cc07dd7e441a"
server_key_version: "encryptionkey@0"
I20260621 07:04:43.690815 9439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:43.711040 9439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:43.716625 9439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:43.716928 9439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:43.720072 9439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:43.720191 9439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:43.720355 9439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:43.720450 9439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:04:43 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:43.745962 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:43.720727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:58465 (local address 127.1.188.254:33173)
0621 07:04:43.723814 (+ 3087us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:43.723821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:43.723856 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:04:43.725705 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:43.725712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:43.725836 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:43.726002 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:43.728361 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.729451 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.731191 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.731807 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.731974 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:43.737718 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:43.737727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:43.737734 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:43.737793 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:43.741540 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.742618 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.742624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.742628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.742817 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.743544 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.743548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.743550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.743962 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:43.744131 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:43.745291 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0621 07:04:43.745561 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2913,"thread_start_us":2767,"threads_started":1}
I20260621 07:04:43.748014 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:43.721600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.193:58465)
0621 07:04:43.722816 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:43.722886 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:43.725334 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:43.726304 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:43.726322 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:43.727176 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:43.728159 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:43.728181 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.729625 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.729643 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:43.731018 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:43.731030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.732091 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.732101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:43.732499 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:43.733518 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:43.733558 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:43.737429 (+ 3871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:43.741698 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:43.741708 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:43.741727 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:43.742474 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:43.742996 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:43.743002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:43.743006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:43.743354 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:43.744210 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:43.744217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:43.744823 (+ 606us) client_negotiation.cc:769] Sending connection context
0621 07:04:43.745178 (+ 355us) client_negotiation.cc:240] Negotiation successful
0621 07:04:43.745825 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":481,"threads_started":1}
I20260621 07:04:43.771938 9439 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:38287
I20260621 07:04:43.771956 9574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:38287 every 8 connection(s)
I20260621 07:04:43.774010 9439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:43.776011 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 9439
I20260621 07:04:43.776294 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:43.776726 9575 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33173
I20260621 07:04:43.777067 9575 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:43.777532 1779 external_mini_cluster.cc:1428] Setting key 59ed07a546831dad321bc7f29004db85
I20260621 07:04:43.778205 9575 heartbeater.cc:511] Master 127.1.188.254:33173 requested a full tablet report, sending...
I20260621 07:04:43.781167 9367 ts_manager.cc:194] Registered new tserver with Master: aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287)
I20260621 07:04:43.783738 9367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:58465
I20260621 07:04:43.788955 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:43.756686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.193:45137)
0621 07:04:43.772023 (+ 15337us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:43.772056 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:43.772737 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:43.773218 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:43.773225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:43.773921 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:43.774417 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:43.774435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.776034 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.776039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:43.777787 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:43.777805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.779118 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.779125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:43.779321 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:43.780293 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:43.780318 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:43.781493 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:43.785764 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:43.785770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:43.785773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:43.786371 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:43.786801 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:43.786806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:43.786809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:43.786966 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:43.787679 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:43.787684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:43.787905 (+ 221us) client_negotiation.cc:769] Sending connection context
0621 07:04:43.788080 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:04:43.788474 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15096,"thread_start_us":15567,"threads_started":1}
I20260621 07:04:43.789283 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:43.756595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45137 (local address 127.1.188.254:33173)
0621 07:04:43.757118 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:43.757127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:43.772223 (+ 15096us) server_negotiation.cc:407] Connection header received
0621 07:04:43.772926 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:43.772931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:43.773033 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:43.773137 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:43.774616 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.775845 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.777979 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:43.778688 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:43.778829 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:43.781720 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:43.781730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:43.781734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:43.781774 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:43.785600 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.786517 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.786523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.786526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.786674 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:43.787090 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:43.787094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:43.787097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:43.787462 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:43.787617 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:43.788669 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0621 07:04:43.788915 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
WARNING: no policy specified for oryx/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
I20260621 07:04:43.842468 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33173
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:44.215263 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:44.215706 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:44.215904 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:44.224071 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:44.224208 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:44.224428 9579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:04:44.224542 9579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:44.235420 9579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:44.237133 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:44.238752 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:44.254768 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:44.254899 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:44.255687 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:44.255756 9579 server_base.cc:1034] running on GCE node
I20260621 07:04:44.256595 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:44.258116 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:44.259384 9579 hybrid_clock.cc:648] HybridClock initialized: now 1782025484259304 us; error 72 us; skew 500 ppm
I20260621 07:04:44.259794 9579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:44 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:44.263522 9579 init.cc:377] Logged in from keytab as oryx/127.1.188.194@KRBTEST.COM (short username oryx)
I20260621 07:04:44.265229 9579 webserver.cc:460] Webserver started at http://127.1.188.194:33787/ using document root <none> and password file <none>
I20260621 07:04:44.266168 9579 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:44.266304 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:44.266659 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:44.270056 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "915927ae9293456badae78acc1563a3a"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "6dc1650f63c679b014b5d08e85535b11"
server_key_iv: "735a4c59ed865a6a84dd7db00d3a9431"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.270864 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "915927ae9293456badae78acc1563a3a"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "6dc1650f63c679b014b5d08e85535b11"
server_key_iv: "735a4c59ed865a6a84dd7db00d3a9431"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.276925 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260621 07:04:44.280956 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:44.282470 9579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:44.282648 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "915927ae9293456badae78acc1563a3a"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "6dc1650f63c679b014b5d08e85535b11"
server_key_iv: "735a4c59ed865a6a84dd7db00d3a9431"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.282912 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:44.306578 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:44.311054 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:44.311312 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:44.314114 9579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:44.314240 9579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:44.314388 9579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:44.314459 9579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:04:44 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:44.341265 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.315115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:44713 (local address 127.1.188.254:33173)
0621 07:04:44.315734 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:44.315739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:44.317009 (+ 1270us) server_negotiation.cc:407] Connection header received
0621 07:04:44.319257 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:44.319263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:44.319378 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:44.319540 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:44.322057 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.323617 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.326335 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.327181 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.327358 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.333206 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:44.333212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:44.333225 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:44.333262 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:44.337141 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.338152 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.338160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.338165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.338339 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.339020 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.339023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.339025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.339451 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:44.339632 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:44.340740 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0621 07:04:44.340943 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":358,"threads_started":1}
I20260621 07:04:44.343592 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.315883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.194:44713)
0621 07:04:44.316777 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:44.316816 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:44.318945 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:44.319859 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:44.319890 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:44.320792 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:44.321795 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.321835 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.323833 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.323845 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.326175 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.326188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.327662 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.327669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.327900 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.328891 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:44.328947 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:44.332914 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:44.337290 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.337301 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.337325 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.337924 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.338467 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.338472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.338476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.338857 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.339631 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:44.339645 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:44.340255 (+ 610us) client_negotiation.cc:769] Sending connection context
0621 07:04:44.340683 (+ 428us) client_negotiation.cc:240] Negotiation successful
0621 07:04:44.341436 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":401,"threads_started":1}
I20260621 07:04:44.368547 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.350044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.194:43897)
0621 07:04:44.350851 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:44.350884 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:44.351071 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:44.351488 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:44.351507 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:44.352444 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:44.353055 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.353074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.354459 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.354469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.356875 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.356890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.357734 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.357744 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.358007 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.358985 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:44.359007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:44.360213 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:44.365021 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.365030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.365059 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.365875 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.366294 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.366300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.366303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.366517 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.367195 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:44.367200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:44.367503 (+ 303us) client_negotiation.cc:769] Sending connection context
0621 07:04:44.367705 (+ 202us) client_negotiation.cc:240] Negotiation successful
0621 07:04:44.368124 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":382,"threads_started":1}
I20260621 07:04:44.369076 9579 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:44973
I20260621 07:04:44.369133 9714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:44973 every 8 connection(s)
I20260621 07:04:44.369755 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.350040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:43897 (local address 127.1.188.254:33173)
0621 07:04:44.350270 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:44.350279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:44.350986 (+ 707us) server_negotiation.cc:407] Connection header received
0621 07:04:44.351255 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:44.351259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:44.351325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:44.351452 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:44.353231 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.354268 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.357049 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.357498 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.357667 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.360533 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:44.360539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:44.360542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:44.360576 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:44.364792 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.366004 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.366010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.366014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.366176 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.366644 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.366649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.366653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.367072 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:44.367242 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:44.369162 (+ 1920us) server_negotiation.cc:299] Negotiation successful
0621 07:04:44.369481 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:04:44.370762 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:44.373669 9715 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33173
I20260621 07:04:44.374059 9715 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:44.374850 9715 heartbeater.cc:511] Master 127.1.188.254:33173 requested a full tablet report, sending...
I20260621 07:04:44.376493 9367 ts_manager.cc:194] Registered new tserver with Master: 915927ae9293456badae78acc1563a3a (127.1.188.194:44973)
I20260621 07:04:44.377413 9367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:44713
I20260621 07:04:44.378427 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 9579
I20260621 07:04:44.378645 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:44.379155 1779 external_mini_cluster.cc:1428] Setting key 47eb4f2549ec539a3e9ffaa4af79713b
WARNING: no policy specified for oryx/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
I20260621 07:04:44.434878 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33173
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:44.788170 9575 heartbeater.cc:503] Master 127.1.188.254:33173 was elected leader, sending a full tablet report...
W20260621 07:04:44.804983 9719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:44.805320 9719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:44.805480 9719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:44.812678 9719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:44.812829 9719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:44.812990 9719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 07:04:44.813155 9719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:44.822952 9719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.188.254:33173
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:44.824553 9719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:44.826102 9719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:44.839946 9725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:44.840019 9727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:44.840610 9724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:44.840885 9719 server_base.cc:1034] running on GCE node
I20260621 07:04:44.841589 9719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:44.842633 9719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:44.843853 9719 hybrid_clock.cc:648] HybridClock initialized: now 1782025484843786 us; error 41 us; skew 500 ppm
I20260621 07:04:44.844213 9719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:44 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:44.847735 9719 init.cc:377] Logged in from keytab as oryx/127.1.188.195@KRBTEST.COM (short username oryx)
I20260621 07:04:44.849220 9719 webserver.cc:460] Webserver started at http://127.1.188.195:36283/ using document root <none> and password file <none>
I20260621 07:04:44.850025 9719 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:44.850127 9719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:44.850469 9719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:44.853508 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "3c34fd3f6ad24baebc52fed800a68391"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "459b955dc5bcd95383137c0c2c7c2d0c"
server_key_iv: "e3110725fa4e286793716cc161ab1020"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.854285 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "3c34fd3f6ad24baebc52fed800a68391"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "459b955dc5bcd95383137c0c2c7c2d0c"
server_key_iv: "e3110725fa4e286793716cc161ab1020"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.859826 9719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 07:04:44.863600 9734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:44.864991 9719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:44.865269 9719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "3c34fd3f6ad24baebc52fed800a68391"
format_stamp: "Formatted at 2026-06-21 07:04:44 on dist-test-slave-hlq1"
server_key: "459b955dc5bcd95383137c0c2c7c2d0c"
server_key_iv: "e3110725fa4e286793716cc161ab1020"
server_key_version: "encryptionkey@0"
I20260621 07:04:44.865504 9719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:44.890978 9719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:44.895371 9719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:44.895706 9719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:44.898404 9719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:44.898520 9719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:44.898672 9719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:44.898753 9719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:04:44 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:44.924880 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.899289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:38573 (local address 127.1.188.254:33173)
0621 07:04:44.899770 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:44.899777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:44.901097 (+ 1320us) server_negotiation.cc:407] Connection header received
0621 07:04:44.903537 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:44.903552 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:44.903632 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:44.903743 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:44.906299 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.907698 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.910185 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.910814 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.910957 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.916424 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:44.916434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:44.916443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:44.916501 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:44.920649 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.921487 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.921496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.921502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.921689 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.922278 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.922286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.922291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.922591 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:44.922759 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:44.923895 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0621 07:04:44.924221 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260621 07:04:44.925571 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.899995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.195:38573)
0621 07:04:44.900906 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:44.900961 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:44.903080 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:44.904349 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:44.904364 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:44.905238 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:44.906038 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.906089 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.907919 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.907925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.910013 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.910029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.911086 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.911094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.911307 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.912189 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:44.912224 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:44.916003 (+ 3779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:44.920839 (+ 4836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.920851 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.920876 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.921355 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.921836 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.921844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.921850 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.922145 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.922734 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:44.922742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:44.923296 (+ 554us) client_negotiation.cc:769] Sending connection context
0621 07:04:44.923735 (+ 439us) client_negotiation.cc:240] Negotiation successful
0621 07:04:44.924331 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":396,"threads_started":1}
I20260621 07:04:44.948191 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.932068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.1.188.195:35037)
0621 07:04:44.932803 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:44.932834 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:44.933006 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:44.933347 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:44.933354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:44.934064 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:44.934524 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.934539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.935610 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.935617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.937393 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:44.937415 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.938642 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.938647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:44.938777 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.939534 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:44.939551 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:44.940802 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:44.945027 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.945032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.945037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.945630 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.946042 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:44.946047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:44.946051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:44.946180 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:44.947028 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:44.947031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:44.947241 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 07:04:44.947357 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:04:44.947819 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":373,"threads_started":1}
I20260621 07:04:44.948309 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:44.932056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:35037 (local address 127.1.188.254:33173)
0621 07:04:44.932271 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:44.932279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:44.932937 (+ 658us) server_negotiation.cc:407] Connection header received
0621 07:04:44.933127 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:44.933132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:44.933203 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:44.933282 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:44.934698 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.935483 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.937557 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:44.938203 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:44.938362 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:44.941052 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:44.941061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:44.941066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:44.941117 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:44.944880 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.945774 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.945781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.945785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.945921 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:44.946340 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:44.946349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:44.946355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:44.946899 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:44.947096 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:44.947528 (+ 432us) server_negotiation.cc:299] Negotiation successful
0621 07:04:44.947798 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:04:44.956259 9719 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:46051
I20260621 07:04:44.956301 9854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:46051 every 8 connection(s)
I20260621 07:04:44.957504 9719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:44.959869 9855 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33173
I20260621 07:04:44.960120 9855 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:44.960863 9855 heartbeater.cc:511] Master 127.1.188.254:33173 requested a full tablet report, sending...
I20260621 07:04:44.962114 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 9719
I20260621 07:04:44.962306 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:44.962435 9367 ts_manager.cc:194] Registered new tserver with Master: 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:44.962836 1779 external_mini_cluster.cc:1428] Setting key 6fb1bf77ef96f379a939562606560726
I20260621 07:04:44.963371 9367 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:38573
I20260621 07:04:44.966053 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 07:04:44 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 07:04:45 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:45 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:45.380717 9715 heartbeater.cc:503] Master 127.1.188.254:33173 was elected leader, sending a full tablet report...
I20260621 07:04:45.966331 9855 heartbeater.cc:503] Master 127.1.188.254:33173 was elected leader, sending a full tablet report...
W20260621 07:04:46.096825 9863 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.084s user 0.004s sys 0.012s
W20260621 07:04:46.098052 9743 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:45.007147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.1.188.254:33173)
0621 07:04:45.007416 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:45.007423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:45.007534 (+ 111us) server_negotiation.cc:407] Connection header received
0621 07:04:45.007742 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:45.007746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:45.007814 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:45.007902 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:45.009193 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:45.009893 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:45.011284 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:45.011750 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:45.011849 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.097776 (+1085927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38010: BlockingRecv error: recv got EOF from 127.0.0.1:38010 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 21 07:04:46 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025484, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:46.123342 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.102242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.1.188.254:33173)
0621 07:04:46.102476 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.102485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.102756 (+ 271us) server_negotiation.cc:407] Connection header received
0621 07:04:46.102982 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.102988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.103067 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.103176 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:46.104561 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.105295 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.106558 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.107037 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.107181 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.117295 (+ 10114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:46.117301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:46.117305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:46.117347 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:46.120576 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:46.121396 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:46.121408 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:46.121410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:46.121538 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:46.121977 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:46.121981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:46.121983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:46.122328 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:46.122526 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.122900 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.123079 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:04:46.128007 9367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:46.136001 9367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:46.159690 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.149327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44973 (local address 127.0.0.1:60960)
0621 07:04:46.149956 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.149994 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.150144 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.152197 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.152204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.152235 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.152634 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.152647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.154577 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.154583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.156520 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.156534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.158452 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.158461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.158674 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.158726 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.158875 (+ 149us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.159210 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":27,"thread_start_us":273,"threads_started":1}
I20260621 07:04:46.160009 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.149513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.1.188.194:44973)
0621 07:04:46.151427 (+ 1914us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.151438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.151469 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:04:46.151584 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.151592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.151912 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.152869 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.153096 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.154405 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.156656 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.158231 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.158364 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.158587 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.158941 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.159690 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1716,"thread_start_us":1519,"threads_started":1}
I20260621 07:04:46.160750 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.150580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:46051 (local address 127.0.0.1:58694)
0621 07:04:46.151662 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.151738 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.151883 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.153035 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.153042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.153070 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.153530 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.153541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.155440 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.155445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.157517 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.157531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.159217 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.159222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.159382 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.159452 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.160177 (+ 725us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.160314 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":47,"spinlock_wait_cycles":22144,"thread_start_us":819,"threads_started":1}
I20260621 07:04:46.161742 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.151150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38287 (local address 127.0.0.1:54220)
0621 07:04:46.153126 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.153148 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.153291 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.153956 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.153960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.153984 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.154320 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.154330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.156770 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.156775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.158886 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.158900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.160752 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.160759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.161053 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.161093 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.161211 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.161337 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":1731,"threads_started":1}
I20260621 07:04:46.161753 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.151225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.1.188.195:46051)
0621 07:04:46.152429 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.152438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.152464 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:04:46.152549 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.152555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.152810 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.153702 (+ 892us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.153910 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.155262 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.157678 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.158997 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.160976 (+ 1979us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.161176 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.161244 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.161450 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":876,"threads_started":1}
I20260621 07:04:46.161772 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.151645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.1.188.193:38287)
0621 07:04:46.152726 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.152736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.153406 (+ 670us) server_negotiation.cc:407] Connection header received
0621 07:04:46.153528 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.153535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.153756 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.153951 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.154493 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.156634 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.159072 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.160489 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.160658 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.161027 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.161263 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.161468 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":758,"threads_started":1}
I20260621 07:04:46.164669 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 4df956d660cf455484d5f01cae7410df (DEFAULT_TABLE table=test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.165107 9785 tablet_service.cc:1467] Processing CreateTablet for tablet 4df956d660cf455484d5f01cae7410df (DEFAULT_TABLE table=test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.165423 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 4df956d660cf455484d5f01cae7410df (DEFAULT_TABLE table=test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.166273 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df956d660cf455484d5f01cae7410df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.166395 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df956d660cf455484d5f01cae7410df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.166662 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df956d660cf455484d5f01cae7410df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.178843 9880 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Bootstrap starting.
I20260621 07:04:46.181193 9882 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Bootstrap starting.
I20260621 07:04:46.184208 9880 tablet_bootstrap.cc:654] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.184801 9882 tablet_bootstrap.cc:654] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.185626 9880 log.cc:824] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:46.185958 9882 log.cc:824] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:46.187201 9881 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Bootstrap starting.
I20260621 07:04:46.189931 9880 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: No bootstrap required, opened a new log
I20260621 07:04:46.190423 9880 ts_tablet_manager.cc:1397] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260621 07:04:46.193032 9882 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: No bootstrap required, opened a new log
I20260621 07:04:46.193341 9882 ts_tablet_manager.cc:1397] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260621 07:04:46.193907 9881 tablet_bootstrap.cc:654] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.195093 9881 log.cc:824] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:46.201278 9881 tablet_bootstrap.cc:492] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: No bootstrap required, opened a new log
I20260621 07:04:46.201557 9881 ts_tablet_manager.cc:1397] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260621 07:04:46.201550 9880 raft_consensus.cc:348] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.202036 9880 raft_consensus.cc:374] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.202147 9880 raft_consensus.cc:729] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.202919 9880 consensus_queue.cc:260] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.203962 9882 raft_consensus.cc:348] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.204502 9882 raft_consensus.cc:374] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.204622 9882 raft_consensus.cc:729] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c34fd3f6ad24baebc52fed800a68391, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.205368 9882 consensus_queue.cc:260] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.207744 9882 ts_tablet_manager.cc:1428] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260621 07:04:46.209290 9880 ts_tablet_manager.cc:1428] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
W20260621 07:04:46.209973 9856 tablet.cc:2369] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:46.211213 9881 raft_consensus.cc:348] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.211681 9881 raft_consensus.cc:374] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.211788 9881 raft_consensus.cc:729] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915927ae9293456badae78acc1563a3a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.212422 9881 consensus_queue.cc:260] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.214115 9881 ts_tablet_manager.cc:1428] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260621 07:04:46.229198 9886 raft_consensus.cc:482] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:46.229564 9886 raft_consensus.cc:504] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.231396 9886 leader_election.cc:288] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.239133 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.232374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:50329 (local address 127.1.188.194:44973)
0621 07:04:46.232569 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.232575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.233138 (+ 563us) server_negotiation.cc:407] Connection header received
0621 07:04:46.233461 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.233465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.233525 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.233600 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.234417 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.235143 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.237218 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.238088 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.238213 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.238269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.238730 (+ 461us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.238878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260621 07:04:46.239183 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.231725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44973 (local address 127.1.188.193:50329)
0621 07:04:46.233027 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.233055 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.233268 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.233696 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.233701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.233761 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.234255 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.234270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.235383 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.235391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.237057 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.237069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.238329 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.238337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.238517 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.238551 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.238647 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.238792 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":45,"thread_start_us":1039,"threads_started":1}
I20260621 07:04:46.239698 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.232129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:58575 (local address 127.1.188.195:46051)
0621 07:04:46.232365 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.232371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.233243 (+ 872us) server_negotiation.cc:407] Connection header received
0621 07:04:46.233445 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.233449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.233524 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.233658 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.234478 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.235180 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.237453 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.238530 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.238704 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.238771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.239295 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.239459 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:04:46.240310 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.231754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:46051 (local address 127.1.188.193:58575)
0621 07:04:46.233161 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.233176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.233282 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.233699 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.233704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.233730 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.234274 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.234284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.235494 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.235498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.237300 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.237313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.238832 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.238839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.239019 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.239054 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.239884 (+ 830us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.240020 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":24,"spinlock_wait_cycles":3328,"thread_start_us":880,"threads_started":1}
I20260621 07:04:46.240288 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a" is_pre_election: true
I20260621 07:04:46.240899 9665 raft_consensus.cc:2454] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:46.241187 9888 raft_consensus.cc:482] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:46.241314 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:46.241436 9888 raft_consensus.cc:504] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.241743 9456 leader_election.cc:302] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915927ae9293456badae78acc1563a3a, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:46.241868 9805 raft_consensus.cc:2454] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:46.242192 9886 raft_consensus.cc:2781] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:46.242347 9886 raft_consensus.cc:482] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:46.242470 9886 raft_consensus.cc:3037] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.243417 9888 leader_election.cc:288] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051), aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287)
I20260621 07:04:46.245993 9886 raft_consensus.cc:504] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.246763 9886 leader_election.cc:288] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Requested vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.247186 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a"
I20260621 07:04:46.247368 9665 raft_consensus.cc:3037] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.248255 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391"
I20260621 07:04:46.248471 9805 raft_consensus.cc:3037] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.251051 9665 raft_consensus.cc:2454] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:46.251649 9456 leader_election.cc:302] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915927ae9293456badae78acc1563a3a, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:46.251989 9886 raft_consensus.cc:2781] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:46.253057 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.244402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:36893 (local address 127.1.188.193:38287)
0621 07:04:46.244671 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.244679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.244703 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:04:46.244778 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.244785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.244869 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.244957 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.246407 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.247525 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.250487 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.251874 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.252002 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.252072 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.252679 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.252825 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:04:46.253615 9805 raft_consensus.cc:2454] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:46.253890 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.243674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38287 (local address 127.1.188.194:36893)
0621 07:04:46.244438 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.244458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.244579 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.245675 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.245681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.245775 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.246201 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.246212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.247667 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.247672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.250310 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.250325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.252304 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.252310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.252479 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.252511 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.253281 (+ 770us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.253451 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":70,"spinlock_wait_cycles":39808,"thread_start_us":1404,"threads_started":1}
I20260621 07:04:46.254005 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.244068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:52379 (local address 127.1.188.195:46051)
0621 07:04:46.245296 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.245300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.245313 (+ 13us) server_negotiation.cc:407] Connection header received
0621 07:04:46.245356 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.245359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.245449 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.245526 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:04:46.246325 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.247432 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.250492 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.252371 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.252552 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.252675 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.253408 (+ 733us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.253610 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133}
I20260621 07:04:46.254815 9892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.243693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:46051 (local address 127.1.188.194:52379)
0621 07:04:46.245113 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.245130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.245230 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.245574 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.245579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.245605 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:04:46.246119 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.246133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.247623 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.247628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.250321 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.250335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.252911 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.252921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.253173 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.253227 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.254206 (+ 979us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.254414 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1303,"mutex_wait_us":79,"thread_start_us":1138,"threads_started":1}
I20260621 07:04:46.254900 9525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" is_pre_election: true
I20260621 07:04:46.255467 9886 raft_consensus.cc:686] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Becoming Leader. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Running, Role: LEADER
I20260621 07:04:46.255442 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df956d660cf455484d5f01cae7410df" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:46.255755 9805 raft_consensus.cc:2379] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 915927ae9293456badae78acc1563a3a in current term 1: Already voted for candidate aaf37f2ba46d43828fd871dd6f7e388a in this term.
I20260621 07:04:46.255923 9886 consensus_queue.cc:237] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } }
I20260621 07:04:46.260049 9596 leader_election.cc:302] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 915927ae9293456badae78acc1563a3a; no voters: 3c34fd3f6ad24baebc52fed800a68391, aaf37f2ba46d43828fd871dd6f7e388a
I20260621 07:04:46.260437 9888 raft_consensus.cc:2726] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 07:04:46.261389 9367 catalog_manager.cc:5495] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193). New cstate: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } health_report { overall_health: HEALTHY } } }
I20260621 07:04:46.289893 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.284574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.1.188.193:38287)
0621 07:04:46.284791 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.284796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.284817 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:04:46.284924 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.284928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.285002 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.285161 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:46.285544 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.286266 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.287767 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.288156 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.288299 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.288732 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.289289 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.289446 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.289504 (+ 58us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.289585 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260621 07:04:46.290663 9576 tablet.cc:2369] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:46.298352 9805 raft_consensus.cc:1261] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.298612 9665 raft_consensus.cc:1261] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.299362 9893 consensus_queue.cc:1035] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:46.299804 9886 consensus_queue.cc:1035] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:46.325769 9368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:46.326020 9368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:46.329967 9368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Sending DeleteTablet for 3 replicas of tablet 4df956d660cf455484d5f01cae7410df
I20260621 07:04:46.331135 9785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df956d660cf455484d5f01cae7410df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:46 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58694
I20260621 07:04:46.331146 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df956d660cf455484d5f01cae7410df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:46 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:54220
I20260621 07:04:46.331939 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df956d660cf455484d5f01cae7410df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:46 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:60960
I20260621 07:04:46.332700 9906 tablet_replica.cc:330] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: stopping tablet replica
I20260621 07:04:46.332896 9907 tablet_replica.cc:330] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: stopping tablet replica
I20260621 07:04:46.333427 9906 raft_consensus.cc:2229] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:46.333316 9368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:04:46.334218 9907 raft_consensus.cc:2229] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260621 07:04:46.334249 9368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:46.334848 9907 raft_consensus.cc:2258] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:46.335695 9906 raft_consensus.cc:2258] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:46.336233 9908 tablet_replica.cc:330] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: stopping tablet replica
I20260621 07:04:46.336917 9908 raft_consensus.cc:2229] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:46.337396 9908 raft_consensus.cc:2258] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:46.339006 9907 ts_tablet_manager.cc:1905] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:46.340816 9906 ts_tablet_manager.cc:1905] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:46.343353 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 19bb388ea6a142448e88fc58b5367769 (DEFAULT_TABLE table=test-table [id=f199cb9e77de43ee91d178c5c1258a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.343618 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 19bb388ea6a142448e88fc58b5367769 (DEFAULT_TABLE table=test-table [id=f199cb9e77de43ee91d178c5c1258a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.343986 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bb388ea6a142448e88fc58b5367769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.344079 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bb388ea6a142448e88fc58b5367769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.347769 9785 tablet_service.cc:1467] Processing CreateTablet for tablet 19bb388ea6a142448e88fc58b5367769 (DEFAULT_TABLE table=test-table [id=f199cb9e77de43ee91d178c5c1258a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:46.348268 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19bb388ea6a142448e88fc58b5367769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.350819 9880 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Bootstrap starting.
I20260621 07:04:46.352586 9880 tablet_bootstrap.cc:654] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.353302 9906 ts_tablet_manager.cc:1918] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:46.353559 9906 log.cc:1196] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/4df956d660cf455484d5f01cae7410df
I20260621 07:04:46.354260 9906 ts_tablet_manager.cc:1939] T 4df956d660cf455484d5f01cae7410df P aaf37f2ba46d43828fd871dd6f7e388a: Deleting consensus metadata
I20260621 07:04:46.356979 9353 catalog_manager.cc:4841] TS aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287): tablet 4df956d660cf455484d5f01cae7410df (table test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]) successfully deleted
I20260621 07:04:46.358261 9881 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Bootstrap starting.
I20260621 07:04:46.358698 9882 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Bootstrap starting.
I20260621 07:04:46.360168 9881 tablet_bootstrap.cc:654] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.360456 9882 tablet_bootstrap.cc:654] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.363188 9880 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: No bootstrap required, opened a new log
I20260621 07:04:46.363338 9880 ts_tablet_manager.cc:1397] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260621 07:04:46.364269 9880 raft_consensus.cc:348] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.364499 9880 raft_consensus.cc:374] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.364603 9880 raft_consensus.cc:729] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.364920 9880 consensus_queue.cc:260] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.367540 9908 ts_tablet_manager.cc:1905] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:46.367735 9880 ts_tablet_manager.cc:1428] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:04:46.373955 9881 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: No bootstrap required, opened a new log
I20260621 07:04:46.375813 9881 ts_tablet_manager.cc:1397] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20260621 07:04:46.376832 9881 raft_consensus.cc:348] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.377094 9881 raft_consensus.cc:374] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.377219 9881 raft_consensus.cc:729] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915927ae9293456badae78acc1563a3a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.377619 9881 consensus_queue.cc:260] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.378435 9881 ts_tablet_manager.cc:1428] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:04:46.381631 9882 tablet_bootstrap.cc:492] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: No bootstrap required, opened a new log
I20260621 07:04:46.381872 9882 ts_tablet_manager.cc:1397] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.007s
I20260621 07:04:46.382812 9882 raft_consensus.cc:348] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.383024 9882 raft_consensus.cc:374] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.383129 9882 raft_consensus.cc:729] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c34fd3f6ad24baebc52fed800a68391, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.383502 9882 consensus_queue.cc:260] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.384761 9882 ts_tablet_manager.cc:1428] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 07:04:46.391187 9908 ts_tablet_manager.cc:1918] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:46.391427 9908 log.cc:1196] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/4df956d660cf455484d5f01cae7410df
I20260621 07:04:46.391999 9908 ts_tablet_manager.cc:1939] T 4df956d660cf455484d5f01cae7410df P 915927ae9293456badae78acc1563a3a: Deleting consensus metadata
I20260621 07:04:46.392941 9907 ts_tablet_manager.cc:1918] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:46.393110 9907 log.cc:1196] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/4df956d660cf455484d5f01cae7410df
I20260621 07:04:46.393615 9907 ts_tablet_manager.cc:1939] T 4df956d660cf455484d5f01cae7410df P 3c34fd3f6ad24baebc52fed800a68391: Deleting consensus metadata
I20260621 07:04:46.393914 9353 catalog_manager.cc:4841] TS 915927ae9293456badae78acc1563a3a (127.1.188.194:44973): tablet 4df956d660cf455484d5f01cae7410df (table test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]) successfully deleted
I20260621 07:04:46.395658 9352 catalog_manager.cc:4841] TS 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051): tablet 4df956d660cf455484d5f01cae7410df (table test-table [id=18c2c8833eeb4fbe8434903ba2b81c7c]) successfully deleted
W20260621 07:04:46.624430 9716 tablet.cc:2369] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:04:46.664698 9888 raft_consensus.cc:482] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:46.664927 9888 raft_consensus.cc:504] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.665737 9888 leader_election.cc:288] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.666405 9525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bb388ea6a142448e88fc58b5367769" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" is_pre_election: true
I20260621 07:04:46.666446 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bb388ea6a142448e88fc58b5367769" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:46.666669 9805 raft_consensus.cc:2454] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915927ae9293456badae78acc1563a3a in term 0.
I20260621 07:04:46.666730 9525 raft_consensus.cc:2454] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915927ae9293456badae78acc1563a3a in term 0.
I20260621 07:04:46.667114 9595 leader_election.cc:302] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c34fd3f6ad24baebc52fed800a68391, 915927ae9293456badae78acc1563a3a; no voters:
I20260621 07:04:46.667474 9888 raft_consensus.cc:2781] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:46.667603 9888 raft_consensus.cc:482] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:46.667712 9888 raft_consensus.cc:3037] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.670990 9888 raft_consensus.cc:504] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.671757 9888 leader_election.cc:288] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 election: Requested vote from peers aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.672241 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bb388ea6a142448e88fc58b5367769" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391"
I20260621 07:04:46.672274 9525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19bb388ea6a142448e88fc58b5367769" candidate_uuid: "915927ae9293456badae78acc1563a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf37f2ba46d43828fd871dd6f7e388a"
I20260621 07:04:46.672456 9805 raft_consensus.cc:3037] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.672493 9525 raft_consensus.cc:3037] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.674798 9525 raft_consensus.cc:2454] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915927ae9293456badae78acc1563a3a in term 1.
I20260621 07:04:46.675194 9596 leader_election.cc:302] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915927ae9293456badae78acc1563a3a, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:46.675501 9888 raft_consensus.cc:2781] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:46.675535 9805 raft_consensus.cc:2454] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915927ae9293456badae78acc1563a3a in term 1.
I20260621 07:04:46.684896 9888 raft_consensus.cc:686] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 1 LEADER]: Becoming Leader. State: Replica: 915927ae9293456badae78acc1563a3a, State: Running, Role: LEADER
I20260621 07:04:46.685328 9888 consensus_queue.cc:237] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.689112 9368 catalog_manager.cc:5495] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 915927ae9293456badae78acc1563a3a (127.1.188.194). New cstate: current_term: 1 leader_uuid: "915927ae9293456badae78acc1563a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } health_report { overall_health: UNKNOWN } } }
Jun 21 07:04:46 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025483, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.254@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:46.753281 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.737963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.1.188.254:33173)
0621 07:04:46.738652 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.738657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.740047 (+ 1390us) server_negotiation.cc:407] Connection header received
0621 07:04:46.740263 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.740269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.740334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.740444 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:46.741532 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.742317 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.743612 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.744022 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.744169 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.747796 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:46.747801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:46.747804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:46.747832 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:46.750522 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:46.751239 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:46.751244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:46.751249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:46.751358 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:46.751753 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:46.751758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:46.751761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:46.752030 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:46.752189 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.752597 (+ 408us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.752836 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":558,"thread_start_us":410,"threads_started":1}
I20260621 07:04:46.753310 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.737960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33173 (local address 127.0.0.1:38026)
0621 07:04:46.739940 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.739958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.740133 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.740482 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.740488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.741192 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:46.741387 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.741396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.742450 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.742457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.743448 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.743465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.744192 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.744197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.744360 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.745148 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:46.745170 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:46.747597 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:46.750653 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:46.750666 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:46.750674 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:46.751111 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:46.751508 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:46.751513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:46.751516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:46.751644 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:46.752175 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:46.752182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:46.752348 (+ 166us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.752511 (+ 163us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.752953 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"mutex_wait_us":161,"thread_start_us":1696,"threads_started":1}
I20260621 07:04:46.760066 9368 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:38026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 07:04:46.761425 9368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:46.769079 9645 tablet_service.cc:1467] Processing CreateTablet for tablet d1f8c44cbc0542fda3ff3bda8632ae9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d3e6ded1d044979b50d173b28f2decf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:04:46.769075 9505 tablet_service.cc:1467] Processing CreateTablet for tablet d1f8c44cbc0542fda3ff3bda8632ae9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d3e6ded1d044979b50d173b28f2decf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:04:46.769164 9785 tablet_service.cc:1467] Processing CreateTablet for tablet d1f8c44cbc0542fda3ff3bda8632ae9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d3e6ded1d044979b50d173b28f2decf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 07:04:46.769657 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f8c44cbc0542fda3ff3bda8632ae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.769671 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f8c44cbc0542fda3ff3bda8632ae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.769699 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f8c44cbc0542fda3ff3bda8632ae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:46.776836 9880 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: Bootstrap starting.
I20260621 07:04:46.776858 9881 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: Bootstrap starting.
I20260621 07:04:46.778621 9880 tablet_bootstrap.cc:654] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.778609 9881 tablet_bootstrap.cc:654] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.779696 9882 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: Bootstrap starting.
I20260621 07:04:46.780980 9880 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: No bootstrap required, opened a new log
I20260621 07:04:46.781042 9882 tablet_bootstrap.cc:654] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:46.781283 9880 ts_tablet_manager.cc:1397] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260621 07:04:46.782266 9880 raft_consensus.cc:348] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.782747 9880 raft_consensus.cc:374] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.782903 9880 raft_consensus.cc:729] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.783321 9880 consensus_queue.cc:260] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.784015 9901 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.784363 9901 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.787698 9882 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: No bootstrap required, opened a new log
I20260621 07:04:46.787887 9882 ts_tablet_manager.cc:1397] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260621 07:04:46.787935 9880 ts_tablet_manager.cc:1428] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260621 07:04:46.788646 9882 raft_consensus.cc:348] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.788913 9882 raft_consensus.cc:374] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.789001 9882 raft_consensus.cc:729] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c34fd3f6ad24baebc52fed800a68391, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.789253 9882 consensus_queue.cc:260] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.789785 9882 ts_tablet_manager.cc:1428] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 07:04:46.789680 9887 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.789978 9887 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.790683 9881 tablet_bootstrap.cc:492] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: No bootstrap required, opened a new log
I20260621 07:04:46.790863 9881 ts_tablet_manager.cc:1397] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260621 07:04:46.791569 9881 raft_consensus.cc:348] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.792151 9881 raft_consensus.cc:374] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:46.792236 9881 raft_consensus.cc:729] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915927ae9293456badae78acc1563a3a, State: Initialized, Role: FOLLOWER
I20260621 07:04:46.792479 9881 consensus_queue.cc:260] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.793023 9881 ts_tablet_manager.cc:1428] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260621 07:04:46.792943 9888 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.793241 9888 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.845402 9901 raft_consensus.cc:482] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:46.845641 9901 raft_consensus.cc:504] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.846345 9901 leader_election.cc:288] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.846886 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f8c44cbc0542fda3ff3bda8632ae9d" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:46.846909 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f8c44cbc0542fda3ff3bda8632ae9d" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a" is_pre_election: true
I20260621 07:04:46.847214 9665 raft_consensus.cc:2454] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:46.847213 9805 raft_consensus.cc:2454] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:46.847648 9455 leader_election.cc:302] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c34fd3f6ad24baebc52fed800a68391, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:46.847890 9901 raft_consensus.cc:2781] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:46.848008 9901 raft_consensus.cc:482] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:46.848119 9901 raft_consensus.cc:3037] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.850073 9901 raft_consensus.cc:504] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.850634 9901 leader_election.cc:288] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Requested vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:46.851042 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f8c44cbc0542fda3ff3bda8632ae9d" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a"
I20260621 07:04:46.851260 9665 raft_consensus.cc:3037] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.851384 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f8c44cbc0542fda3ff3bda8632ae9d" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391"
I20260621 07:04:46.851625 9805 raft_consensus.cc:3037] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:46.854238 9665 raft_consensus.cc:2454] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:46.854352 9805 raft_consensus.cc:2454] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:46.854693 9456 leader_election.cc:302] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915927ae9293456badae78acc1563a3a, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:46.855010 9901 raft_consensus.cc:2781] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:46.855544 9901 raft_consensus.cc:686] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Becoming Leader. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Running, Role: LEADER
I20260621 07:04:46.855850 9901 consensus_queue.cc:237] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:46.857475 9924 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: TxnStatusTablet state changed. Reason: New leader aaf37f2ba46d43828fd871dd6f7e388a. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.857751 9924 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:04:46.858523 9926 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 07:04:46.858824 9926 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 07:04:46.859036 9367 catalog_manager.cc:5495] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193). New cstate: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } health_report { overall_health: UNKNOWN } } }
I20260621 07:04:46.893236 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.887271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.1.188.193:38287)
0621 07:04:46.888018 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.888022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.888038 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:04:46.888098 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.888101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.888163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.888244 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:46.888691 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.889471 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.891385 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.891896 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.892001 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.892520 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.892652 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.892718 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.892883 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.892942 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":420,"threads_started":1}
I20260621 07:04:46.893328 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.887139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38287 (local address 127.0.0.1:54244)
0621 07:04:46.887350 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.887370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.887540 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.888308 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.888314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.888327 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:04:46.888520 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.888531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.889611 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.889616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.891229 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.891244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.892045 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.892051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.892280 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.892351 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.892751 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.892772 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.892860 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.893010 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:04:46.897732 9665 raft_consensus.cc:1261] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.897856 9805 raft_consensus.cc:1261] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.898346 9924 consensus_queue.cc:1035] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:46.898717 9901 consensus_queue.cc:1035] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:46.902194 9887 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: TxnStatusTablet state changed. Reason: New leader aaf37f2ba46d43828fd871dd6f7e388a. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.902534 9887 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.906721 9888 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: TxnStatusTablet state changed. Reason: New leader aaf37f2ba46d43828fd871dd6f7e388a. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.906991 9888 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.908138 9931 mvcc.cc:205] Tried to move back new op lower bound from 7299176394327183360 to 7299176394164174848. Current Snapshot: MvccSnapshot[applied={T|T < 7299176394327183360}]
I20260621 07:04:46.909884 9924 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.910163 9924 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:04:46.911685 9887 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.911936 9887 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 3c34fd3f6ad24baebc52fed800a68391: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.915977 9901 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.916328 9901 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P aaf37f2ba46d43828fd871dd6f7e388a: This TxnStatusTablet replica's current role is: LEADER
I20260621 07:04:46.923804 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.916551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.1.188.194:44973)
0621 07:04:46.916997 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.917003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.917021 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:04:46.917119 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.917124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.917221 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.917398 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:46.917927 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.920045 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.921423 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.921883 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.922029 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.922475 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.923041 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.923217 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.923333 (+ 116us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.923445 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":2495,"threads_started":1}
I20260621 07:04:46.927955 9888 tablet_replica.cc:439] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } } }
I20260621 07:04:46.928294 9888 tablet_replica.cc:442] T d1f8c44cbc0542fda3ff3bda8632ae9d P 915927ae9293456badae78acc1563a3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 07:04:46.940691 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.934268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38287 (local address 127.1.188.194:50933)
0621 07:04:46.934835 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:46.934863 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:46.934978 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:46.935288 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:46.935291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:46.935300 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:04:46.935634 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.935642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.937131 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.937135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.938227 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:46.938239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.939361 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.939372 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:46.939594 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.939660 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.940200 (+ 540us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.940211 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:04:46.940286 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:04:46.940393 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":369,"threads_started":1}
I20260621 07:04:46.940696 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:46.934565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:50933 (local address 127.1.188.193:38287)
0621 07:04:46.934777 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:46.934784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:46.934940 (+ 156us) server_negotiation.cc:407] Connection header received
0621 07:04:46.935077 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:46.935082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:46.935169 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:46.935270 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:46.935792 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.936981 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.938368 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:46.939049 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:46.939181 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:46.939859 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:46.939995 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:46.940114 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:46.940332 (+ 218us) server_negotiation.cc:299] Negotiation successful
0621 07:04:46.940416 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:04:46.951311 9805 raft_consensus.cc:1261] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Refusing update from remote peer 915927ae9293456badae78acc1563a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.951597 9525 raft_consensus.cc:1261] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Refusing update from remote peer 915927ae9293456badae78acc1563a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:46.952386 9912 consensus_queue.cc:1035] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:46.952724 9888 consensus_queue.cc:1035] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:47.007093 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.000433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:33837 (local address 127.1.188.194:44973)
0621 07:04:47.000632 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.000639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.001148 (+ 509us) server_negotiation.cc:407] Connection header received
0621 07:04:47.001357 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.001363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.001443 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.001539 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:47.002720 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.003513 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.005401 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.005805 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.005931 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.006369 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:47.006470 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:47.006566 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.006695 (+ 129us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.006772 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 07:04:47.007222 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.000231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44973 (local address 127.1.188.193:33837)
0621 07:04:47.001038 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:47.001055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:47.001212 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:47.001554 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:47.001559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:47.001570 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:04:47.002505 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:47.002522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.003904 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.003911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:47.005252 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:47.005264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.005972 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.005981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:47.006128 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.006178 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:04:47.006569 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:04:47.006583 (+ 14us) client_negotiation.cc:769] Sending connection context
0621 07:04:47.006675 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:04:47.006822 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":421,"threads_started":1}
I20260621 07:04:47.059355 9367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.059697 9367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.063330 9367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Sending DeleteTablet for 3 replicas of tablet 19bb388ea6a142448e88fc58b5367769
I20260621 07:04:47.064347 9785 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bb388ea6a142448e88fc58b5367769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58694
I20260621 07:04:47.065156 9504 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bb388ea6a142448e88fc58b5367769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:54220
I20260621 07:04:47.065836 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 19bb388ea6a142448e88fc58b5367769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:60960
I20260621 07:04:47.080338 9952 tablet_replica.cc:330] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: stopping tablet replica
I20260621 07:04:47.080569 9952 raft_consensus.cc:2229] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.080839 9952 raft_consensus.cc:2258] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.082091 9952 ts_tablet_manager.cc:1905] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.089321 9952 ts_tablet_manager.cc:1918] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.089458 9952 log.cc:1196] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/19bb388ea6a142448e88fc58b5367769
I20260621 07:04:47.089895 9952 ts_tablet_manager.cc:1939] T 19bb388ea6a142448e88fc58b5367769 P 3c34fd3f6ad24baebc52fed800a68391: Deleting consensus metadata
I20260621 07:04:47.090999 9352 catalog_manager.cc:4841] TS 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051): tablet 19bb388ea6a142448e88fc58b5367769 (table test-table [id=f199cb9e77de43ee91d178c5c1258a9e]) successfully deleted
I20260621 07:04:47.097325 9954 tablet_replica.cc:330] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: stopping tablet replica
I20260621 07:04:47.097708 9954 raft_consensus.cc:2229] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:47.098371 9953 tablet_replica.cc:330] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: stopping tablet replica
I20260621 07:04:47.098997 9954 raft_consensus.cc:2258] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.099318 9953 raft_consensus.cc:2229] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.100692 9953 raft_consensus.cc:2258] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.101429 9954 ts_tablet_manager.cc:1905] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.102545 9953 ts_tablet_manager.cc:1905] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.115254 9953 ts_tablet_manager.cc:1918] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.115432 9953 log.cc:1196] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/19bb388ea6a142448e88fc58b5367769
I20260621 07:04:47.115896 9953 ts_tablet_manager.cc:1939] T 19bb388ea6a142448e88fc58b5367769 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting consensus metadata
I20260621 07:04:47.117126 9353 catalog_manager.cc:4841] TS aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287): tablet 19bb388ea6a142448e88fc58b5367769 (table test-table [id=f199cb9e77de43ee91d178c5c1258a9e]) successfully deleted
I20260621 07:04:47.122525 9954 ts_tablet_manager.cc:1918] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.122679 9954 log.cc:1196] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/19bb388ea6a142448e88fc58b5367769
I20260621 07:04:47.123210 9954 ts_tablet_manager.cc:1939] T 19bb388ea6a142448e88fc58b5367769 P 915927ae9293456badae78acc1563a3a: Deleting consensus metadata
I20260621 07:04:47.124506 9353 catalog_manager.cc:4841] TS 915927ae9293456badae78acc1563a3a (127.1.188.194:44973): tablet 19bb388ea6a142448e88fc58b5367769 (table test-table [id=f199cb9e77de43ee91d178c5c1258a9e]) successfully deleted
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:04:47.168884 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:47.154389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.1.188.254:33173)
0621 07:04:47.154621 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.154628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.155178 (+ 550us) server_negotiation.cc:407] Connection header received
0621 07:04:47.155349 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.155355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.155471 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.155574 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:47.157299 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.158496 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.160450 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.161139 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.161313 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.168563 (+ 7250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38038: BlockingRecv error: recv got EOF from 127.0.0.1:38038 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM
I20260621 07:04:47.192233 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.173066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38042 (local address 127.1.188.254:33173)
0621 07:04:47.173239 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.173245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.173438 (+ 193us) server_negotiation.cc:407] Connection header received
0621 07:04:47.173687 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.173693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.173777 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.173862 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:47.175607 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.176747 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.178694 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.179388 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.179549 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.185016 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:47.185023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:47.185028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:47.185077 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:47.188778 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.189656 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.189664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.189668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.189856 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.190318 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.190324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.190328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.190692 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:47.190943 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.191583 (+ 640us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.191854 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 07:04:47.197283 9367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:47.197991 9367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:47.206215 9504 tablet_service.cc:1467] Processing CreateTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 (DEFAULT_TABLE table=test-table [id=343125977ba440abb3dcdd2e499ca371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.206324 9785 tablet_service.cc:1467] Processing CreateTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 (DEFAULT_TABLE table=test-table [id=343125977ba440abb3dcdd2e499ca371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.206722 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d21734773247f6b23a69e0e6f8dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.206774 9504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d21734773247f6b23a69e0e6f8dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.212075 9645 tablet_service.cc:1467] Processing CreateTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 (DEFAULT_TABLE table=test-table [id=343125977ba440abb3dcdd2e499ca371]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.212524 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d21734773247f6b23a69e0e6f8dfa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.213774 9880 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Bootstrap starting.
I20260621 07:04:47.215507 9880 tablet_bootstrap.cc:654] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.223171 9882 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Bootstrap starting.
I20260621 07:04:47.225479 9882 tablet_bootstrap.cc:654] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.228050 9880 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: No bootstrap required, opened a new log
I20260621 07:04:47.228227 9880 ts_tablet_manager.cc:1397] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260621 07:04:47.229259 9880 raft_consensus.cc:348] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.229537 9881 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Bootstrap starting.
I20260621 07:04:47.229565 9880 raft_consensus.cc:374] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.229660 9880 raft_consensus.cc:729] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.229938 9880 consensus_queue.cc:260] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.231768 9881 tablet_bootstrap.cc:654] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.233238 9880 ts_tablet_manager.cc:1428] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260621 07:04:47.237723 9881 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: No bootstrap required, opened a new log
I20260621 07:04:47.237947 9881 ts_tablet_manager.cc:1397] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260621 07:04:47.238451 9882 tablet_bootstrap.cc:492] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: No bootstrap required, opened a new log
I20260621 07:04:47.238607 9882 ts_tablet_manager.cc:1397] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20260621 07:04:47.239389 9882 raft_consensus.cc:348] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.239688 9882 raft_consensus.cc:374] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.239789 9882 raft_consensus.cc:729] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c34fd3f6ad24baebc52fed800a68391, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.239807 9881 raft_consensus.cc:348] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.240031 9882 consensus_queue.cc:260] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.240691 9882 ts_tablet_manager.cc:1428] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 07:04:47.240298 9881 raft_consensus.cc:374] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.240990 9881 raft_consensus.cc:729] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915927ae9293456badae78acc1563a3a, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.241271 9881 consensus_queue.cc:260] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.241860 9881 ts_tablet_manager.cc:1428] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 07:04:47.255255 9943 raft_consensus.cc:482] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:47.255532 9943 raft_consensus.cc:504] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.256369 9943 leader_election.cc:288] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:47.256976 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d21734773247f6b23a69e0e6f8dfa4" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a" is_pre_election: true
I20260621 07:04:47.257247 9665 raft_consensus.cc:2454] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:47.257745 9456 leader_election.cc:302] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915927ae9293456badae78acc1563a3a, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:47.258136 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d21734773247f6b23a69e0e6f8dfa4" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:47.258419 9805 raft_consensus.cc:2454] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:47.258709 9943 raft_consensus.cc:2781] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:47.258864 9943 raft_consensus.cc:482] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:47.258966 9943 raft_consensus.cc:3037] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.261415 9943 raft_consensus.cc:504] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.262048 9943 leader_election.cc:288] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Requested vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:47.262504 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d21734773247f6b23a69e0e6f8dfa4" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a"
I20260621 07:04:47.262732 9665 raft_consensus.cc:3037] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.262748 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d21734773247f6b23a69e0e6f8dfa4" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391"
I20260621 07:04:47.262934 9805 raft_consensus.cc:3037] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.266016 9805 raft_consensus.cc:2454] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:47.266498 9455 leader_election.cc:302] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c34fd3f6ad24baebc52fed800a68391, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:47.266505 9665 raft_consensus.cc:2454] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:47.267005 9943 raft_consensus.cc:2781] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:47.267242 9943 raft_consensus.cc:686] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Becoming Leader. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Running, Role: LEADER
I20260621 07:04:47.267618 9943 consensus_queue.cc:237] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.270874 9366 catalog_manager.cc:5495] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193). New cstate: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } health_report { overall_health: UNKNOWN } } }
I20260621 07:04:47.300122 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.294620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.1.188.193:38287)
0621 07:04:47.295146 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.295150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.295170 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:04:47.295227 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.295231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.295321 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.295462 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:47.295949 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.296805 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.298209 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.298670 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.298815 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.299233 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:47.299363 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:47.299555 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.299708 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.299792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411}
I20260621 07:04:47.303079 9665 raft_consensus.cc:1261] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:47.303864 9805 raft_consensus.cc:1261] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:47.303881 9943 consensus_queue.cc:1035] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:47.304562 9901 consensus_queue.cc:1035] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:47.318917 9946 mvcc.cc:205] Tried to move back new op lower bound from 7299176395987714048 to 7299176395851300864. Current Snapshot: MvccSnapshot[applied={T|T < 7299176395987714048}]
I20260621 07:04:47.330987 9367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.331336 9367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.334726 9367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Sending DeleteTablet for 3 replicas of tablet a7d21734773247f6b23a69e0e6f8dfa4
I20260621 07:04:47.335791 9504 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:54220
I20260621 07:04:47.335861 9785 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58694
I20260621 07:04:47.336129 9953 tablet_replica.cc:330] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: stopping tablet replica
I20260621 07:04:47.336122 9952 tablet_replica.cc:330] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: stopping tablet replica
I20260621 07:04:47.336326 9953 raft_consensus.cc:2229] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:47.336341 9952 raft_consensus.cc:2229] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.336426 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d21734773247f6b23a69e0e6f8dfa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:60960
I20260621 07:04:47.336555 9952 raft_consensus.cc:2258] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.336642 9953 raft_consensus.cc:2258] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.336948 9954 tablet_replica.cc:330] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: stopping tablet replica
I20260621 07:04:47.337159 9954 raft_consensus.cc:2229] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.337385 9954 raft_consensus.cc:2258] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.338527 9954 ts_tablet_manager.cc:1905] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.338780 9367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:04:47.339442 9367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:04:47.347012 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 (DEFAULT_TABLE table=test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.347007 9785 tablet_service.cc:1467] Processing CreateTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 (DEFAULT_TABLE table=test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.347467 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80200cb9ba6a4e17b040919ea11fd806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.347467 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80200cb9ba6a4e17b040919ea11fd806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.348045 9952 ts_tablet_manager.cc:1905] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.351444 9504 tablet_service.cc:1467] Processing CreateTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 (DEFAULT_TABLE table=test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:04:47.351949 9504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80200cb9ba6a4e17b040919ea11fd806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:47.354974 9952 ts_tablet_manager.cc:1918] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:47.355106 9952 log.cc:1196] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/a7d21734773247f6b23a69e0e6f8dfa4
I20260621 07:04:47.355429 9882 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Bootstrap starting.
I20260621 07:04:47.355563 9952 ts_tablet_manager.cc:1939] T a7d21734773247f6b23a69e0e6f8dfa4 P 3c34fd3f6ad24baebc52fed800a68391: Deleting consensus metadata
I20260621 07:04:47.356550 9953 ts_tablet_manager.cc:1905] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.357221 9882 tablet_bootstrap.cc:654] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.360019 9352 catalog_manager.cc:4841] TS 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051): tablet a7d21734773247f6b23a69e0e6f8dfa4 (table test-table [id=343125977ba440abb3dcdd2e499ca371]) successfully deleted
I20260621 07:04:47.365474 9954 ts_tablet_manager.cc:1918] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:47.365661 9954 log.cc:1196] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/a7d21734773247f6b23a69e0e6f8dfa4
I20260621 07:04:47.367938 9953 ts_tablet_manager.cc:1918] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:04:47.368108 9953 log.cc:1196] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/a7d21734773247f6b23a69e0e6f8dfa4
I20260621 07:04:47.370136 9881 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Bootstrap starting.
I20260621 07:04:47.370708 9954 ts_tablet_manager.cc:1939] T a7d21734773247f6b23a69e0e6f8dfa4 P 915927ae9293456badae78acc1563a3a: Deleting consensus metadata
I20260621 07:04:47.371929 9953 ts_tablet_manager.cc:1939] T a7d21734773247f6b23a69e0e6f8dfa4 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting consensus metadata
I20260621 07:04:47.372296 9881 tablet_bootstrap.cc:654] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.373651 9353 catalog_manager.cc:4841] TS 915927ae9293456badae78acc1563a3a (127.1.188.194:44973): tablet a7d21734773247f6b23a69e0e6f8dfa4 (table test-table [id=343125977ba440abb3dcdd2e499ca371]) successfully deleted
I20260621 07:04:47.374392 9353 catalog_manager.cc:4841] TS aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287): tablet a7d21734773247f6b23a69e0e6f8dfa4 (table test-table [id=343125977ba440abb3dcdd2e499ca371]) successfully deleted
I20260621 07:04:47.375968 9880 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Bootstrap starting.
I20260621 07:04:47.377803 9880 tablet_bootstrap.cc:654] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:47.380023 9881 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: No bootstrap required, opened a new log
I20260621 07:04:47.380234 9881 ts_tablet_manager.cc:1397] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260621 07:04:47.381170 9881 raft_consensus.cc:348] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.381484 9881 raft_consensus.cc:374] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.381620 9881 raft_consensus.cc:729] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915927ae9293456badae78acc1563a3a, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.381924 9881 consensus_queue.cc:260] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.382781 9882 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: No bootstrap required, opened a new log
I20260621 07:04:47.382979 9882 ts_tablet_manager.cc:1397] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20260621 07:04:47.383697 9882 raft_consensus.cc:348] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.383929 9882 raft_consensus.cc:374] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.384038 9882 raft_consensus.cc:729] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c34fd3f6ad24baebc52fed800a68391, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.384322 9882 consensus_queue.cc:260] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.385484 9881 ts_tablet_manager.cc:1428] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260621 07:04:47.389076 9882 ts_tablet_manager.cc:1428] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260621 07:04:47.400691 9880 tablet_bootstrap.cc:492] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: No bootstrap required, opened a new log
I20260621 07:04:47.400920 9880 ts_tablet_manager.cc:1397] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20260621 07:04:47.401746 9880 raft_consensus.cc:348] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.402005 9880 raft_consensus.cc:374] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:47.402081 9880 raft_consensus.cc:729] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Initialized, Role: FOLLOWER
I20260621 07:04:47.403659 9880 consensus_queue.cc:260] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.404943 9880 ts_tablet_manager.cc:1428] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260621 07:04:47.508170 9943 raft_consensus.cc:482] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:04:47.508476 9943 raft_consensus.cc:504] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.509254 9943 leader_election.cc:288] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:47.509900 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80200cb9ba6a4e17b040919ea11fd806" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a" is_pre_election: true
I20260621 07:04:47.509939 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80200cb9ba6a4e17b040919ea11fd806" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391" is_pre_election: true
I20260621 07:04:47.510200 9805 raft_consensus.cc:2454] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:47.510208 9665 raft_consensus.cc:2454] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 0.
I20260621 07:04:47.510699 9455 leader_election.cc:302] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c34fd3f6ad24baebc52fed800a68391, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:47.510986 9943 raft_consensus.cc:2781] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:04:47.511137 9943 raft_consensus.cc:482] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:04:47.511222 9943 raft_consensus.cc:3037] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.514259 9943 raft_consensus.cc:504] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.515058 9943 leader_election.cc:288] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Requested vote from peers 915927ae9293456badae78acc1563a3a (127.1.188.194:44973), 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051)
I20260621 07:04:47.515614 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80200cb9ba6a4e17b040919ea11fd806" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915927ae9293456badae78acc1563a3a"
I20260621 07:04:47.515702 9805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80200cb9ba6a4e17b040919ea11fd806" candidate_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c34fd3f6ad24baebc52fed800a68391"
I20260621 07:04:47.515899 9805 raft_consensus.cc:3037] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.515892 9665 raft_consensus.cc:3037] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:47.518908 9805 raft_consensus.cc:2454] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:47.519133 9665 raft_consensus.cc:2454] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf37f2ba46d43828fd871dd6f7e388a in term 1.
I20260621 07:04:47.519372 9455 leader_election.cc:302] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c34fd3f6ad24baebc52fed800a68391, aaf37f2ba46d43828fd871dd6f7e388a; no voters:
I20260621 07:04:47.519665 9943 raft_consensus.cc:2781] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:47.519831 9943 raft_consensus.cc:686] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Becoming Leader. State: Replica: aaf37f2ba46d43828fd871dd6f7e388a, State: Running, Role: LEADER
I20260621 07:04:47.520121 9943 consensus_queue.cc:237] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } }
I20260621 07:04:47.522888 9367 catalog_manager.cc:5495] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193). New cstate: current_term: 1 leader_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf37f2ba46d43828fd871dd6f7e388a" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 38287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 } health_report { overall_health: UNKNOWN } } }
I20260621 07:04:47.553529 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.547919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:49329 (local address 127.1.188.193:38287)
0621 07:04:47.548149 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.548155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.548407 (+ 252us) server_negotiation.cc:407] Connection header received
0621 07:04:47.548576 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.548579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.548638 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.548729 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:04:47.549150 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.549848 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.551853 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.552340 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.552451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.552888 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:04:47.552989 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:04:47.553065 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.553217 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.553279 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:04:47.553818 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.547802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:38287 (local address 127.1.188.193:49329)
0621 07:04:47.548306 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:47.548327 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:47.548468 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:47.548770 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:47.548775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:47.548795 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 07:04:47.549017 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:47.549029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.549981 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.549987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:47.551694 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:47.551709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.552500 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.552507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:47.552670 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.552717 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 07:04:47.553091 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 07:04:47.553103 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 07:04:47.553189 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 07:04:47.553345 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":3,"thread_start_us":221,"threads_started":1}
I20260621 07:04:47.562124 9805 raft_consensus.cc:1261] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:47.562357 9665 raft_consensus.cc:1261] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Refusing update from remote peer aaf37f2ba46d43828fd871dd6f7e388a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:04:47.562788 9982 consensus_queue.cc:1035] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c34fd3f6ad24baebc52fed800a68391" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:47.563136 9981 consensus_queue.cc:1035] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [LEADER]: Connected to new peer: Peer: permanent_uuid: "915927ae9293456badae78acc1563a3a" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:04:47.636108 9366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.636329 9366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:04:47.638996 9366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P deda2fd32dbb414ca4956ca7ac38c46c: Sending DeleteTablet for 3 replicas of tablet 80200cb9ba6a4e17b040919ea11fd806
I20260621 07:04:47.639722 9504 tablet_service.cc:1514] Processing DeleteTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:54220
I20260621 07:04:47.640019 9953 tablet_replica.cc:330] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: stopping tablet replica
I20260621 07:04:47.640123 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:60960
I20260621 07:04:47.640202 9953 raft_consensus.cc:2229] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:04:47.640153 9785 tablet_service.cc:1514] Processing DeleteTablet for tablet 80200cb9ba6a4e17b040919ea11fd806 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:04:47 UTC) from {username='oryx', principal='oryx/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58694
I20260621 07:04:47.640398 9954 tablet_replica.cc:330] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: stopping tablet replica
I20260621 07:04:47.640450 9952 tablet_replica.cc:330] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: stopping tablet replica
I20260621 07:04:47.640547 9953 raft_consensus.cc:2258] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.640566 9954 raft_consensus.cc:2229] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.640618 9952 raft_consensus.cc:2229] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:04:47.640748 9954 raft_consensus.cc:2258] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.640831 9952 raft_consensus.cc:2258] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:04:47.641516 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 9439
I20260621 07:04:47.641790 9953 ts_tablet_manager.cc:1905] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.642030 9954 ts_tablet_manager.cc:1905] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.644735 9952 ts_tablet_manager.cc:1905] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:04:47.648381 9954 ts_tablet_manager.cc:1918] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.648408 9953 ts_tablet_manager.cc:1918] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.648517 9954 log.cc:1196] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/80200cb9ba6a4e17b040919ea11fd806
I20260621 07:04:47.648518 9953 log.cc:1196] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/80200cb9ba6a4e17b040919ea11fd806
I20260621 07:04:47.648960 9954 ts_tablet_manager.cc:1939] T 80200cb9ba6a4e17b040919ea11fd806 P 915927ae9293456badae78acc1563a3a: Deleting consensus metadata
I20260621 07:04:47.648960 9953 ts_tablet_manager.cc:1939] T 80200cb9ba6a4e17b040919ea11fd806 P aaf37f2ba46d43828fd871dd6f7e388a: Deleting consensus metadata
I20260621 07:04:47.649942 9353 catalog_manager.cc:4841] TS 915927ae9293456badae78acc1563a3a (127.1.188.194:44973): tablet 80200cb9ba6a4e17b040919ea11fd806 (table test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]) successfully deleted
I20260621 07:04:47.650238 9353 catalog_manager.cc:4841] TS aaf37f2ba46d43828fd871dd6f7e388a (127.1.188.193:38287): tablet 80200cb9ba6a4e17b040919ea11fd806 (table test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]) successfully deleted
I20260621 07:04:47.652372 9952 ts_tablet_manager.cc:1918] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 07:04:47.652518 9952 log.cc:1196] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/80200cb9ba6a4e17b040919ea11fd806
I20260621 07:04:47.652904 9952 ts_tablet_manager.cc:1939] T 80200cb9ba6a4e17b040919ea11fd806 P 3c34fd3f6ad24baebc52fed800a68391: Deleting consensus metadata
I20260621 07:04:47.653800 9352 catalog_manager.cc:4841] TS 3c34fd3f6ad24baebc52fed800a68391 (127.1.188.195:46051): tablet 80200cb9ba6a4e17b040919ea11fd806 (table test-table [id=09c0d79bd2cb4da3b40a3d36e45369d6]) successfully deleted
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.193@KRBTEST.COM
I20260621 07:04:47.664716 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.642467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.1.188.193:38287)
0621 07:04:47.642627 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.642636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.650553 (+ 7917us) server_negotiation.cc:407] Connection header received
0621 07:04:47.650739 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.650746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.650833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.650939 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:47.652344 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.653199 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.654570 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.655024 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.655162 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.659113 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:47.659126 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:47.659144 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:47.659174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:47.661989 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.662649 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.662656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.662658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.662772 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.663146 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.663155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.663157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.663573 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:47.663865 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.664153 (+ 288us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.664389 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 07:04:47.665428 9570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:47.832461 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 9439
I20260621 07:04:47.862473 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 9579
Jun 21 07:04:47 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.194@KRBTEST.COM
I20260621 07:04:47.879144 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:47.863337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.1.188.194:44973)
0621 07:04:47.863863 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:47.863868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:47.863886 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:04:47.863953 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:47.863957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:47.864035 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:47.864134 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:47.865551 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.866533 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.868033 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:47.868484 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:47.868586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:47.873017 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:47.873035 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:47.873054 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:47.873093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:47.876374 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.877158 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.877166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.877168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.877318 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:47.877648 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:47.877651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:47.877653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:47.878079 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:47.878339 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:47.878667 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 07:04:47.878853 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":301,"threads_started":1}
I20260621 07:04:47.879942 9710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:48.021358 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 9579
I20260621 07:04:48.049682 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 9719
Jun 21 07:04:48 dist-test-slave-hlq1 krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.188.195@KRBTEST.COM
I20260621 07:04:48.067036 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:48.050609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.1.188.195:46051)
0621 07:04:48.051219 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:48.051225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:48.051244 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:04:48.051312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:48.051317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:48.051388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:48.051628 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:48.052978 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:48.053780 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:48.055164 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:48.055647 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:48.055747 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:48.060727 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:48.060745 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:48.060766 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:48.060810 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:48.064154 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:48.064937 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:48.064945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:48.064948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:48.065077 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:48.065439 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:48.065442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:48.065444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:48.065920 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:48.066145 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:48.066486 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 07:04:48.066668 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":324,"threads_started":1}
I20260621 07:04:48.067914 9850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:48.177018 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 9719
I20260621 07:04:48.196946 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 9336
I20260621 07:04:48.198407 9408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:48.329227 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 9336
2026-06-21T07:04:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7011 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:48 dist-test-slave-hlq1 krb5kdc[9995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:48 dist-test-slave-hlq1 krb5kdc[9995](info): set up 2 sockets
Jun 21 07:04:48 dist-test-slave-hlq1 krb5kdc[9995](info): commencing operation
krb5kdc: starting...
W20260621 07:04:49.452849 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:49 dist-test-slave-hlq1 krb5kdc[9995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:49.715539 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40481
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:50.093560 10008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:50.094002 10008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:50.094192 10008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:50.101872 10008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:50.102051 10008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:50.102172 10008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:50.102280 10008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:50.102375 10008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:50.112875 10008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:50.114470 10008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:50.116063 10008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:50.129230 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:50.129132 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:50.129168 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:50.130163 10008 server_base.cc:1034] running on GCE node
I20260621 07:04:50.130990 10008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:50.132625 10008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:50.133867 10008 hybrid_clock.cc:648] HybridClock initialized: now 1782025490133824 us; error 51 us; skew 500 ppm
I20260621 07:04:50.134255 10008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T07:04:50Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1875 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:04:50 dist-test-slave-hlq1 krb5kdc[10022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:04:50 dist-test-slave-hlq1 krb5kdc[10022](info): set up 2 sockets
Jun 21 07:04:50 dist-test-slave-hlq1 krb5kdc[10022](info): commencing operation
krb5kdc: starting...
W20260621 07:04:51.320873 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:04:51 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:04:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:04:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:04:51.575179 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46349
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:51.945971 10035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:51.946385 10035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:51.946569 10035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:51.954164 10035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:51.954321 10035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:51.954433 10035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:51.954546 10035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:51.954632 10035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:51.964958 10035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:51.966526 10035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:51.968068 10035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:51.981257 10041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:51.981340 10040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:51.981300 10043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:51.982395 10035 server_base.cc:1034] running on GCE node
I20260621 07:04:51.983292 10035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:51.985023 10035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:51.986264 10035 hybrid_clock.cc:648] HybridClock initialized: now 1782025491986224 us; error 64 us; skew 500 ppm
I20260621 07:04:51.986651 10035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:51 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:51.993649 10035 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:51.995337 10035 webserver.cc:460] Webserver started at http://127.1.188.254:38481/ using document root <none> and password file <none>
I20260621 07:04:51.996173 10035 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:51.996279 10035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:51.996582 10035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:51.999976 10035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "634a7d2ce8074c1fb4b361d097bddbee"
format_stamp: "Formatted at 2026-06-21 07:04:51 on dist-test-slave-hlq1"
server_key: "1a0d19e34172acfe47824f5849b4a210"
server_key_iv: "0cde673424189c3ae212af9356ff4c11"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.000779 10035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "634a7d2ce8074c1fb4b361d097bddbee"
format_stamp: "Formatted at 2026-06-21 07:04:51 on dist-test-slave-hlq1"
server_key: "1a0d19e34172acfe47824f5849b4a210"
server_key_iv: "0cde673424189c3ae212af9356ff4c11"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.006426 10035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:04:52.010149 10050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:52.011816 10035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:52.012012 10035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "634a7d2ce8074c1fb4b361d097bddbee"
format_stamp: "Formatted at 2026-06-21 07:04:51 on dist-test-slave-hlq1"
server_key: "1a0d19e34172acfe47824f5849b4a210"
server_key_iv: "0cde673424189c3ae212af9356ff4c11"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.012253 10035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:52.034632 10035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:52.039181 10035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:52.039554 10035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:52.060707 10035 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:46349
I20260621 07:04:52.060710 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:46349 every 8 connection(s)
I20260621 07:04:52.062363 10035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:52.066962 10102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:04:52.069334 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10035
I20260621 07:04:52.069576 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:04:52.070439 1779 external_mini_cluster.cc:1428] Setting key 302733c96b5886d46da86572639e883a
I20260621 07:04:52.080298 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Bootstrap starting.
I20260621 07:04:52.083967 10102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Neither blocks nor log segments found. Creating new log.
I20260621 07:04:52.085294 10102 log.cc:824] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:04:52 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:52.088724 10102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: No bootstrap required, opened a new log
I20260621 07:04:52.094362 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:52.072904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.1.188.254:46349)
0621 07:04:52.073714 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:52.073731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:52.073780 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:04:52.075686 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:52.075712 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:52.076144 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:52.076561 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:52.078194 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.079607 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.080971 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.081651 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.081879 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:52.086604 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:52.086617 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:52.086638 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:52.086688 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:52.089876 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:52.090719 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:52.090726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:52.090736 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:52.090905 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:52.091353 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:52.091358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:52.091360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:52.092029 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:52.092287 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:52.092759 (+ 472us) server_negotiation.cc:299] Negotiation successful
0621 07:04:52.093147 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":332,"threads_started":1}
I20260621 07:04:52.097991 10102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:52.098410 10102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:04:52.098517 10102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a7d2ce8074c1fb4b361d097bddbee, State: Initialized, Role: FOLLOWER
I20260621 07:04:52.099177 10102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:52.099483 10102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:52.099650 10102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:52.099825 10102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:04:52.102138 10102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:52.102690 10102 leader_election.cc:302] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634a7d2ce8074c1fb4b361d097bddbee; no voters:
I20260621 07:04:52.103365 10102 leader_election.cc:288] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:04:52.103544 10107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:04:52.104672 10107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 LEADER]: Becoming Leader. State: Replica: 634a7d2ce8074c1fb4b361d097bddbee, State: Running, Role: LEADER
I20260621 07:04:52.105279 10107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:52.105834 10102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:52.108629 10108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634a7d2ce8074c1fb4b361d097bddbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } } }
I20260621 07:04:52.108597 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634a7d2ce8074c1fb4b361d097bddbee. Latest consensus state: current_term: 1 leader_uuid: "634a7d2ce8074c1fb4b361d097bddbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } } }
I20260621 07:04:52.109089 10108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:52.109155 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:52.109965 10114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:52.116309 10114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:52.126869 10114 catalog_manager.cc:1347] Generated new cluster ID: 93a10d5fb8294c7a86e28b648fc4026e
I20260621 07:04:52.127000 10114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:52.148084 10114 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:04:52.150133 10114 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:52.159920 10114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Generated new TSK 0
I20260621 07:04:52.160991 10114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:04:52.238500 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:52.646328 10128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:52.646658 10128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:52.646759 10128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:52.654328 10128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:52.654459 10128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:52.654587 10128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:52.664988 10128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:52.666555 10128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:52.668084 10128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:52.682971 10136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:52.683084 10133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:52.683059 10134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:52.683889 10128 server_base.cc:1034] running on GCE node
I20260621 07:04:52.684618 10128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:52.685725 10128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:52.686981 10128 hybrid_clock.cc:648] HybridClock initialized: now 1782025492686933 us; error 64 us; skew 500 ppm
I20260621 07:04:52.687376 10128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:52 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:52.691104 10128 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:04:52.692782 10128 webserver.cc:460] Webserver started at http://127.1.188.193:34493/ using document root <none> and password file <none>
I20260621 07:04:52.693564 10128 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:52.693677 10128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:52.693986 10128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:52.697395 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "129e23d4af2e403bb8f2606497395d4d"
format_stamp: "Formatted at 2026-06-21 07:04:52 on dist-test-slave-hlq1"
server_key: "58291cba1858e1e7ed87aa807721dcf6"
server_key_iv: "cf0d8951dfdc8eb6f84ac5abaede1721"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.698372 10128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "129e23d4af2e403bb8f2606497395d4d"
format_stamp: "Formatted at 2026-06-21 07:04:52 on dist-test-slave-hlq1"
server_key: "58291cba1858e1e7ed87aa807721dcf6"
server_key_iv: "cf0d8951dfdc8eb6f84ac5abaede1721"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.704193 10128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 07:04:52.708222 10143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:52.709764 10128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260621 07:04:52.709946 10128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "129e23d4af2e403bb8f2606497395d4d"
format_stamp: "Formatted at 2026-06-21 07:04:52 on dist-test-slave-hlq1"
server_key: "58291cba1858e1e7ed87aa807721dcf6"
server_key_iv: "cf0d8951dfdc8eb6f84ac5abaede1721"
server_key_version: "encryptionkey@0"
I20260621 07:04:52.710158 10128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:52.730369 10128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:52.735273 10128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:52.735633 10128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:52.736702 10128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:52.738230 10128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:52.738309 10128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:52.738404 10128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:52.738449 10128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:52.774555 10128 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:46839
I20260621 07:04:52.774593 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:46839 every 8 connection(s)
I20260621 07:04:52.776119 10128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:52.781531 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10128
I20260621 07:04:52.781776 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:04:52.782680 1779 external_mini_cluster.cc:1428] Setting key 720336903272cbcdc7ad80aa5d0bf6dc
Jun 21 07:04:52 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:52.801508 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:52.779193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45957 (local address 127.1.188.254:46349)
0621 07:04:52.779707 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:52.779711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:52.780771 (+ 1060us) server_negotiation.cc:407] Connection header received
0621 07:04:52.783667 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:52.783674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:52.783775 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:52.783903 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:52.785784 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.786536 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.789299 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.789741 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.789844 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:52.794472 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:52.794478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:52.794482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:52.794520 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:52.797796 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:52.798804 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:52.798810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:52.798814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:52.798986 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:52.799527 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:52.799533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:52.799536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:52.799859 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:52.800045 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:52.800840 (+ 795us) server_negotiation.cc:299] Negotiation successful
0621 07:04:52.801118 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":281,"threads_started":1}
I20260621 07:04:52.802711 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:52.779803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:45957)
0621 07:04:52.780577 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:52.780626 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:52.783283 (+ 2657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:52.784163 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:52.784172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:52.784740 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:52.785579 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:52.785607 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.786709 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.786718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:52.789135 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:52.789154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:52.789983 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:52.789992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:52.790331 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:52.791199 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:52.791244 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:52.794226 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:52.797946 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:52.797956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:52.797976 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:52.798638 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:52.799133 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:52.799137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:52.799139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:52.799333 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:52.800009 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:52.800019 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:52.800418 (+ 399us) client_negotiation.cc:769] Sending connection context
0621 07:04:52.800725 (+ 307us) client_negotiation.cc:240] Negotiation successful
0621 07:04:52.801336 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
I20260621 07:04:52.804616 10256 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46349
I20260621 07:04:52.804975 10256 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:52.805968 10256 heartbeater.cc:511] Master 127.1.188.254:46349 requested a full tablet report, sending...
I20260621 07:04:52.808696 10067 ts_manager.cc:194] Registered new tserver with Master: 129e23d4af2e403bb8f2606497395d4d (127.1.188.193:46839)
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:04:52.810479 10067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:45957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:04:52.840378 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:53.188450 10262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:53.188730 10262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:53.188896 10262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:53.195979 10262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:53.196080 10262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:53.196316 10262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:53.205758 10262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:53.207190 10262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:53.208498 10262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:53.222407 10267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:53.222429 10268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:53.222353 10270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:53.223416 10262 server_base.cc:1034] running on GCE node
I20260621 07:04:53.224047 10262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:53.224995 10262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:53.226198 10262 hybrid_clock.cc:648] HybridClock initialized: now 1782025493226153 us; error 36 us; skew 500 ppm
I20260621 07:04:53.226478 10262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:53 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:53.229797 10262 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:04:53.231221 10262 webserver.cc:460] Webserver started at http://127.1.188.194:36435/ using document root <none> and password file <none>
I20260621 07:04:53.231933 10262 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:53.232033 10262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:53.232287 10262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:53.235060 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "e20d138c31a54f1db44b7fd1e1baaa23"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "a7db65b841dbc906f74139f8f37a7db3"
server_key_iv: "bc9961f4d89fbe7b16a4390fc88d9cea"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.235847 10262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "e20d138c31a54f1db44b7fd1e1baaa23"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "a7db65b841dbc906f74139f8f37a7db3"
server_key_iv: "bc9961f4d89fbe7b16a4390fc88d9cea"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.240818 10262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:04:53.244369 10277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.245808 10262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:04:53.245986 10262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "e20d138c31a54f1db44b7fd1e1baaa23"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "a7db65b841dbc906f74139f8f37a7db3"
server_key_iv: "bc9961f4d89fbe7b16a4390fc88d9cea"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.246165 10262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:53.267382 10262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:53.271685 10262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:53.271982 10262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:53.272927 10262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:53.274312 10262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:53.274385 10262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.274554 10262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:53.274605 10262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.309370 10262 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43635
I20260621 07:04:53.309459 10389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43635 every 8 connection(s)
I20260621 07:04:53.310717 10262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:53.314838 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10262
I20260621 07:04:53.315021 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:04:53.315596 1779 external_mini_cluster.cc:1428] Setting key 8df14f926bf1e32cdd6b13d2d9505799
Jun 21 07:04:53 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:53.333076 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:53.313447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:49973 (local address 127.1.188.254:46349)
0621 07:04:53.313948 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:53.313954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:53.314825 (+ 871us) server_negotiation.cc:407] Connection header received
0621 07:04:53.317241 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:53.317247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:53.317343 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:53.317462 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:53.318944 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.319771 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.321301 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.321746 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.321880 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:53.325600 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:53.325610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:53.325618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:53.325666 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:53.329474 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:53.330242 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:53.330251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:53.330257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:53.330421 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:53.330883 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:53.330890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:53.330896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:53.331309 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:53.331507 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:53.332244 (+ 737us) server_negotiation.cc:299] Negotiation successful
0621 07:04:53.332518 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
I20260621 07:04:53.333659 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:53.313858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:49973)
0621 07:04:53.314671 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:53.314707 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:53.316939 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:53.317712 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:53.317720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:53.318219 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:53.318775 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:53.318791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.319913 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.319919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:53.321151 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:53.321163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.321965 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.321974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:53.322210 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:53.322898 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:53.322922 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:53.325370 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:53.329627 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:53.329634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:53.329647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:53.330063 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:53.330555 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:53.330559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:53.330561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:53.330748 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:53.331508 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:53.331514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:53.331856 (+ 342us) client_negotiation.cc:769] Sending connection context
0621 07:04:53.332147 (+ 291us) client_negotiation.cc:240] Negotiation successful
0621 07:04:53.332530 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260621 07:04:53.335168 10390 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46349
I20260621 07:04:53.335572 10390 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:53.336498 10390 heartbeater.cc:511] Master 127.1.188.254:46349 requested a full tablet report, sending...
I20260621 07:04:53.338097 10067 ts_manager.cc:194] Registered new tserver with Master: e20d138c31a54f1db44b7fd1e1baaa23 (127.1.188.194:43635)
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:04:53.339001 10067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:49973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:04:53.366353 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:53.694450 10396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:53.694715 10396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:53.694840 10396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:53.702353 10396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:53.702468 10396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:53.702657 10396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:53.712764 10396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:53.714054 10396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:53.715422 10396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:53.728683 10402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:53.728710 10401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:53.728888 10404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:53.729667 10396 server_base.cc:1034] running on GCE node
I20260621 07:04:53.730259 10396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:53.731158 10396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:53.732368 10396 hybrid_clock.cc:648] HybridClock initialized: now 1782025493732317 us; error 43 us; skew 500 ppm
I20260621 07:04:53.732700 10396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:53 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:53.735992 10396 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:04:53.737596 10396 webserver.cc:460] Webserver started at http://127.1.188.195:38737/ using document root <none> and password file <none>
I20260621 07:04:53.738301 10396 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:53.738442 10396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:53.738768 10396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:04:53.741755 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "1582de3c61d441e8b633513920356f85"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "7b13c4b2741a98fda9701f51c60b61fc"
server_key_iv: "21f4f18d9ff688c09078451c354b46fa"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.742538 10396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "1582de3c61d441e8b633513920356f85"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "7b13c4b2741a98fda9701f51c60b61fc"
server_key_iv: "21f4f18d9ff688c09078451c354b46fa"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.747889 10396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 07:04:53.751652 10411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.752977 10396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:04:53.753154 10396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "1582de3c61d441e8b633513920356f85"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "7b13c4b2741a98fda9701f51c60b61fc"
server_key_iv: "21f4f18d9ff688c09078451c354b46fa"
server_key_version: "encryptionkey@0"
I20260621 07:04:53.753366 10396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:53.787554 10396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:53.791978 10396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:53.792272 10396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:53.793218 10396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:53.794625 10396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:53.794727 10396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.794881 10396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:53.794960 10396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:53.814546 10256 heartbeater.cc:503] Master 127.1.188.254:46349 was elected leader, sending a full tablet report...
I20260621 07:04:53.829294 10396 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:37807
I20260621 07:04:53.829383 10523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:37807 every 8 connection(s)
I20260621 07:04:53.830592 10396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:53.837566 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10396
I20260621 07:04:53.837775 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:04:53.838243 1779 external_mini_cluster.cc:1428] Setting key 5139ee985e30b2d7835a357bec214bd6
Jun 21 07:04:53 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:04:53.851719 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:53.833293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:34865 (local address 127.1.188.254:46349)
0621 07:04:53.833506 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:53.833511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:53.834726 (+ 1215us) server_negotiation.cc:407] Connection header received
0621 07:04:53.836670 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:53.836676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:53.836782 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:53.836893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:53.838482 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.839503 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.841730 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.842134 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.842237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:53.845796 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:53.845801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:53.845805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:53.845836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:53.848510 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:53.849226 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:53.849234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:53.849240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:53.849361 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:53.849858 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:53.849864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:53.849868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:53.850206 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:53.850374 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:53.851158 (+ 784us) server_negotiation.cc:299] Negotiation successful
0621 07:04:53.851375 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 07:04:53.852640 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:53.833748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:34865)
0621 07:04:53.834542 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:53.834608 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:53.836396 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:53.837128 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:53.837137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:53.837652 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:53.838293 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:53.838310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.839706 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.839711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:53.841575 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:53.841589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:53.842350 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:53.842355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:53.842544 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:53.843262 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:53.843289 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:53.845558 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:04:53.848667 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:53.848676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:53.848699 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:53.849098 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:53.849470 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:04:53.849476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:04:53.849480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:04:53.849745 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:04:53.850333 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:04:53.850341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:04:53.850704 (+ 363us) client_negotiation.cc:769] Sending connection context
0621 07:04:53.851011 (+ 307us) client_negotiation.cc:240] Negotiation successful
0621 07:04:53.851447 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":309,"threads_started":1}
I20260621 07:04:53.854228 10524 heartbeater.cc:347] Connected to a master server at 127.1.188.254:46349
I20260621 07:04:53.854488 10524 heartbeater.cc:464] Registering TS with master...
I20260621 07:04:53.855204 10524 heartbeater.cc:511] Master 127.1.188.254:46349 requested a full tablet report, sending...
I20260621 07:04:53.856714 10066 ts_manager.cc:194] Registered new tserver with Master: 1582de3c61d441e8b633513920356f85 (127.1.188.195:37807)
I20260621 07:04:53.857762 10066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:34865
I20260621 07:04:53.866036 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
Entry for principal oryx/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab.
WARNING: no policy specified for oryx/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
Entry for principal oryx/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab.
WARNING: no policy specified for oryx/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
Entry for principal oryx/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab.
I20260621 07:04:54.002264 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10128
Jun 21 07:04:54 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:04:54.020242 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:54.003542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.1.188.193:46839)
0621 07:04:54.004405 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:54.004413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:54.004441 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:04:54.004504 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:54.004510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:54.004729 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:54.004858 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:54.006380 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.007550 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.008815 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.009388 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.009503 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:54.013747 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:54.013761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:54.013812 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:54.013851 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:54.017412 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.018163 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.018170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.018172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.018316 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.018692 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.018695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.018697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.019026 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:54.019275 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:54.019622 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 07:04:54.019814 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":508,"threads_started":1}
I20260621 07:04:54.021549 10251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:54.110160 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10128
I20260621 07:04:54.127211 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10262
Jun 21 07:04:54 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:04:54.143772 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:54.128128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.1.188.194:43635)
0621 07:04:54.128724 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:54.128732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:54.128761 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:04:54.128823 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:54.128829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:54.129042 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:54.129174 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:54.130447 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.131889 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.133242 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.133799 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.133909 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:54.138133 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:54.138145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:54.138160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:54.138204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:54.141158 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.141816 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.141824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.141826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.141959 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.142327 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.142330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.142332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.142676 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:54.142903 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:54.143165 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 07:04:54.143453 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
I20260621 07:04:54.145020 10385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:54.227692 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10262
I20260621 07:04:54.243346 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10396
Jun 21 07:04:54 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:04:54.260432 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:54.244375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.1.188.195:37807)
0621 07:04:54.244927 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:54.244934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:54.244962 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:04:54.245018 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:54.245025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:54.245241 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:54.245409 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:54.246796 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.247978 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.249565 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:54.250103 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:54.250242 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:54.254919 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:54.254929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:54.254942 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:54.254974 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:54.257726 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.258399 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.258405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.258408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.258518 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:54.258878 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:54.258884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:54.258887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:54.259222 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:54.259559 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:54.259858 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 07:04:54.260099 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":254,"threads_started":1}
I20260621 07:04:54.261673 10519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:54.344154 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10396
I20260621 07:04:54.360319 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10035
I20260621 07:04:54.361557 10097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:04:54.466406 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10035
I20260621 07:04:54.487056 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46349
--webserver_interface=127.1.188.254
--webserver_port=38481
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:04:54.807199 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:54.807546 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:54.807693 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:54.814788 10541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:04:54.814888 10541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:54.814987 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:54.815065 10541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:04:54.815150 10541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:04:54.824987 10541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=38481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:54.826321 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:54.827610 10541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:54.839347 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:54.839463 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:54.839519 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:54.840469 10541 server_base.cc:1034] running on GCE node
I20260621 07:04:54.841252 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:54.842540 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:54.843744 10541 hybrid_clock.cc:648] HybridClock initialized: now 1782025494843687 us; error 55 us; skew 500 ppm
I20260621 07:04:54.844043 10541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:54 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:54.847133 10541 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:04:54.848742 10541 webserver.cc:460] Webserver started at http://127.1.188.254:38481/ using document root <none> and password file <none>
I20260621 07:04:54.849390 10541 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:54.849496 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:54.854871 10541 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:04:54.858148 10556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:54.859527 10541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:04:54.859670 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "634a7d2ce8074c1fb4b361d097bddbee"
format_stamp: "Formatted at 2026-06-21 07:04:51 on dist-test-slave-hlq1"
server_key: "1a0d19e34172acfe47824f5849b4a210"
server_key_iv: "0cde673424189c3ae212af9356ff4c11"
server_key_version: "encryptionkey@0"
I20260621 07:04:54.860638 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:54.876874 10541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:54.880895 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:54.881129 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:54.899796 10541 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:46349
I20260621 07:04:54.899813 10607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:46349 every 8 connection(s)
I20260621 07:04:54.901285 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:04:54.908349 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10541
I20260621 07:04:54.909278 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:46839
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:46349
--webserver_port=34493
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:04:54.909875 10608 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:04:54.913566 10608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Bootstrap starting.
I20260621 07:04:54.935534 10608 log.cc:824] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Log is configured to *not* fsync() on all Append() calls
I20260621 07:04:54.945425 10608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:04:54.945998 10608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Bootstrap complete.
I20260621 07:04:54.956017 10608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:54.956920 10608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634a7d2ce8074c1fb4b361d097bddbee, State: Initialized, Role: FOLLOWER
I20260621 07:04:54.957651 10608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:54.957912 10608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:04:54.958039 10608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:04:54.958223 10608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:04:54.962224 10608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:54.962791 10608 leader_election.cc:302] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634a7d2ce8074c1fb4b361d097bddbee; no voters:
I20260621 07:04:54.963531 10608 leader_election.cc:288] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:04:54.963680 10612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:04:54.965250 10612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [term 2 LEADER]: Becoming Leader. State: Replica: 634a7d2ce8074c1fb4b361d097bddbee, State: Running, Role: LEADER
I20260621 07:04:54.965900 10612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } }
I20260621 07:04:54.966166 10608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:04:54.968645 10614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634a7d2ce8074c1fb4b361d097bddbee. Latest consensus state: current_term: 2 leader_uuid: "634a7d2ce8074c1fb4b361d097bddbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } } }
I20260621 07:04:54.969098 10614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:54.969682 10613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "634a7d2ce8074c1fb4b361d097bddbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634a7d2ce8074c1fb4b361d097bddbee" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 46349 } } }
I20260621 07:04:54.970036 10613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee [sys.catalog]: This master's current role is: LEADER
I20260621 07:04:54.972360 10618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:04:54.979738 10618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:04:54.981665 10618 catalog_manager.cc:1259] Loaded cluster ID: 93a10d5fb8294c7a86e28b648fc4026e
I20260621 07:04:54.981853 10618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:04:54.987203 10618 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:04:54.989168 10618 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 634a7d2ce8074c1fb4b361d097bddbee: Loaded TSK: 0
I20260621 07:04:54.991223 10618 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:04:55.321998 10610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:55.322316 10610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:55.322479 10610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:55.330107 10610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:55.330215 10610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:55.330403 10610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:04:55.340529 10610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.193.keytab
--principal=oryx/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:46839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=34493
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:55.342069 10610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:55.343701 10610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:55.360534 10635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:55.360630 10636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:55.361764 10638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:55.362382 10610 server_base.cc:1034] running on GCE node
I20260621 07:04:55.363066 10610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:55.364313 10610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:55.365552 10610 hybrid_clock.cc:648] HybridClock initialized: now 1782025495365480 us; error 54 us; skew 500 ppm
I20260621 07:04:55.365916 10610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:55 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025495, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:55.369745 10610 init.cc:377] Logged in from keytab as oryx/127.1.188.193@KRBTEST.COM (short username oryx)
I20260621 07:04:55.371583 10610 webserver.cc:460] Webserver started at http://127.1.188.193:34493/ using document root <none> and password file <none>
I20260621 07:04:55.372406 10610 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:55.372586 10610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:55.378975 10610 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:04:55.382179 10645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.383842 10610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:04:55.384033 10610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "129e23d4af2e403bb8f2606497395d4d"
format_stamp: "Formatted at 2026-06-21 07:04:52 on dist-test-slave-hlq1"
server_key: "58291cba1858e1e7ed87aa807721dcf6"
server_key_iv: "cf0d8951dfdc8eb6f84ac5abaede1721"
server_key_version: "encryptionkey@0"
I20260621 07:04:55.385280 10610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:55.411831 10610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:55.416734 10610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:55.417114 10610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:55.418306 10610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:55.419864 10610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:55.420002 10610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.420142 10610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:55.420244 10610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.457484 10610 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:46839
I20260621 07:04:55.457579 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:46839 every 8 connection(s)
I20260621 07:04:55.459214 10610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:04:55.467141 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10610
I20260621 07:04:55.468075 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:43635
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:46349
--webserver_port=36435
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:04:55 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:55 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:04:55.852739 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:55.853071 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:55.853241 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:55.860867 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:55.861024 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:55.861204 10762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:04:55.871687 10762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.194.keytab
--principal=oryx/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:43635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=36435
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:55.873209 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:55.874965 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:55.894877 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:55.894939 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:55.895443 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:55.897122 10762 server_base.cc:1034] running on GCE node
I20260621 07:04:55.897814 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:55.899004 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:55.900310 10762 hybrid_clock.cc:648] HybridClock initialized: now 1782025495900209 us; error 94 us; skew 500 ppm
I20260621 07:04:55.900681 10762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:55 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025495, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:55.904269 10762 init.cc:377] Logged in from keytab as oryx/127.1.188.194@KRBTEST.COM (short username oryx)
I20260621 07:04:55.906232 10762 webserver.cc:460] Webserver started at http://127.1.188.194:36435/ using document root <none> and password file <none>
I20260621 07:04:55.907047 10762 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:55.907217 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:55.913242 10762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:04:55.916442 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.918020 10762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:04:55.918218 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "e20d138c31a54f1db44b7fd1e1baaa23"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "a7db65b841dbc906f74139f8f37a7db3"
server_key_iv: "bc9961f4d89fbe7b16a4390fc88d9cea"
server_key_version: "encryptionkey@0"
I20260621 07:04:55.919256 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:55.947644 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:55.952587 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:55.952891 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:55.954093 10762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:55.955745 10762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:55.955881 10762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.956007 10762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:55.956102 10762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:55.998981 10762 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43635
I20260621 07:04:55.999017 10890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43635 every 8 connection(s)
I20260621 07:04:56.000662 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:04:56.008530 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10762
I20260621 07:04:56.009691 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:37807
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:46349
--webserver_port=38737
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:04:56.405195 10895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:04:56.405509 10895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:04:56.405682 10895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:04:56.412854 10895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:04:56.413004 10895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:04:56.413220 10895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:04:56.424526 10895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/krb5kdc/oryx_127.1.188.195.keytab
--principal=oryx/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:37807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=38737
--tserver_master_addrs=127.1.188.254:46349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:04:56.425868 10895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:04:56.427147 10895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:04:56.444485 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:56.447628 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:04:56.448483 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:04:56.448827 10895 server_base.cc:1034] running on GCE node
I20260621 07:04:56.449687 10895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:04:56.450781 10895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:04:56.451997 10895 hybrid_clock.cc:648] HybridClock initialized: now 1782025496451857 us; error 134 us; skew 500 ppm
I20260621 07:04:56.452373 10895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025496, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:04:56.461040 10895 init.cc:377] Logged in from keytab as oryx/127.1.188.195@KRBTEST.COM (short username oryx)
I20260621 07:04:56.473054 10895 webserver.cc:460] Webserver started at http://127.1.188.195:38737/ using document root <none> and password file <none>
I20260621 07:04:56.474085 10895 fs_manager.cc:359] Metadata directory not provided
I20260621 07:04:56.474298 10895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:04:56.480341 10895 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:04:56.483319 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:56.485028 10895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:04:56.485255 10895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "1582de3c61d441e8b633513920356f85"
format_stamp: "Formatted at 2026-06-21 07:04:53 on dist-test-slave-hlq1"
server_key: "7b13c4b2741a98fda9701f51c60b61fc"
server_key_iv: "21f4f18d9ff688c09078451c354b46fa"
server_key_version: "encryptionkey@0"
I20260621 07:04:56.486285 10895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:04:56.549894 10895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:04:56.559445 10895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:04:56.559885 10895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:04:56.561690 10895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:04:56.564653 10895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:04:56.564905 10895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:56.565080 10895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:04:56.565264 10895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:04:56.653904 10895 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:37807
I20260621 07:04:56.653993 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:37807 every 8 connection(s)
I20260621 07:04:56.656795 10895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:04:56.657423 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 10895
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:56.706878 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:56.658875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.1.188.254:46349)
0621 07:04:56.668024 (+ 9149us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:56.668034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:56.679741 (+ 11707us) server_negotiation.cc:407] Connection header received
0621 07:04:56.679821 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:56.679827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:56.679924 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:56.680052 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:56.690252 (+ 10200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.691493 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.693439 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.694033 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.694140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.698550 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:04:56.698563 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:04:56.698588 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:04:56.698644 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:04:56.701707 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:56.702848 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:56.702856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:56.702862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:56.703027 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:04:56.703533 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:04:56.703540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:04:56.703548 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:04:56.703993 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:04:56.704322 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0621 07:04:56.704841 (+ 519us) server_negotiation.cc:299] Negotiation successful
0621 07:04:56.705362 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8916,"thread_start_us":8708,"threads_started":1}
W20260621 07:04:56.813446 10761 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.336s user 0.002s sys 0.015s
W20260621 07:04:56.814839 10760 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:55.462733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60627 (local address 127.1.188.254:46349)
0621 07:04:55.463617 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:55.463645 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:55.463935 (+ 290us) server_negotiation.cc:407] Connection header received
0621 07:04:55.466864 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:55.466901 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:55.467526 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:55.468045 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:55.469899 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:55.472126 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:55.474349 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:55.475559 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:55.475779 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.814513 (+1338734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:60627: BlockingRecv error: recv got EOF from 127.1.188.193:60627 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":322,"threads_started":1}
I20260621 07:04:56.815383 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:55.462752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:60627)
0621 07:04:55.463733 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:55.463779 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:55.466087 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:55.468238 (+ 2151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:55.468252 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:55.468989 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:55.469661 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:55.469682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:55.472346 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:55.472355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:55.474169 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:55.474185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:55.475793 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:55.475799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:55.476019 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:55.477057 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:55.477099 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:56.814257 (+1337158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":421,"threads_started":1}
W20260621 07:04:56.816265 10758 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:46349 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.188.254:46349: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:56.827546 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:56.816992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:52409)
0621 07:04:56.817555 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:56.817581 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:56.817677 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:56.818371 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:56.818375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:56.818760 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:56.818993 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.819005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.820384 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.820389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.821369 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.821380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.822568 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.822574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.822759 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.823544 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:56.823562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:56.827197 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":727,"threads_started":1}
W20260621 07:04:56.827663 10760 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:56.817939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:52409 (local address 127.1.188.254:46349)
0621 07:04:56.818098 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:56.818102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:56.818117 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:04:56.818162 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:56.818165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:56.818221 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:56.818338 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:56.819164 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.820002 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.821515 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.822197 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.822319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.827419 (+ 5100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:52409: BlockingRecv error: recv got EOF from 127.1.188.193:52409 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:56 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:56.837342 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:56.828515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:38789)
0621 07:04:56.828827 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:56.828841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:56.828960 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:56.829484 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:56.829487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:56.829747 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:56.829893 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.829900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.830863 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.830869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.831938 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.831949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.832614 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.832619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.832790 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.833410 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:56.833433 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:56.837064 (+ 3631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223}
W20260621 07:04:56.837502 10760 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:56.828579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:38789 (local address 127.1.188.254:46349)
0621 07:04:56.828725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:56.828729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:56.828910 (+ 181us) server_negotiation.cc:407] Connection header received
0621 07:04:56.829083 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:56.829088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:56.829176 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:56.829295 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:56.830027 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.830723 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.832074 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.832469 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.832602 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.837236 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:38789: BlockingRecv error: recv got EOF from 127.1.188.193:38789 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260621 07:04:56.837982 10758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:04:57.415503 10894 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.401s user 0.003s sys 0.013s
W20260621 07:04:57.417119 10893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:56.003888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:34001 (local address 127.1.188.254:46349)
0621 07:04:56.004385 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:56.004393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:56.006124 (+ 1731us) server_negotiation.cc:407] Connection header received
0621 07:04:56.008189 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:56.008195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:56.008294 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:56.008418 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:56.010142 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.010925 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.012659 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.013107 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.013256 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.416900 (+1403644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:34001: BlockingRecv error: recv got EOF from 127.1.188.194:34001 (error 108)
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":440,"threads_started":1}
I20260621 07:04:57.418409 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:56.005019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:34001)
0621 07:04:56.005957 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:56.006007 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:56.007905 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:56.008710 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:56.008724 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:56.009270 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:56.009907 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.009937 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.011112 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.011118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.012505 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.012518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.013370 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.013380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.013651 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.014398 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:56.014422 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.416787 (+1402365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":396,"threads_started":1}
W20260621 07:04:57.419416 10891 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:46349 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.188.254:46349: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:57.434507 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:57.420156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:40533)
0621 07:04:57.420343 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:57.420368 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:57.420461 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:57.420865 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:57.420868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:57.421196 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:57.421360 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.421368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.423500 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.423504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.424564 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.424580 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.425892 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.425896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.426011 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.430151 (+ 4140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:57.430169 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.434284 (+ 4115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260621 07:04:57.435897 10893 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:57.420226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:40533 (local address 127.1.188.254:46349)
0621 07:04:57.420469 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:57.420478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:57.420505 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:04:57.420582 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:57.420590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:57.420701 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:57.420861 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:57.421513 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.423233 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.424713 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.425688 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.425877 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.434914 (+ 9037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:40533: BlockingRecv error: recv got EOF from 127.1.188.194:40533 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:57.447760 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:57.435306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:35013)
0621 07:04:57.436329 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:57.436358 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:57.436523 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:57.436909 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:57.436917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:57.437500 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:57.437839 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.437856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.438876 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.438884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.441228 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.441250 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.442658 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.442662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.442813 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.443634 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:57.443674 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.447558 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":861}
W20260621 07:04:57.448088 10891 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:04:57.448457 11032 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:57.435450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:35013 (local address 127.1.188.254:46349)
0621 07:04:57.435971 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:57.436000 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:57.436447 (+ 447us) server_negotiation.cc:407] Connection header received
0621 07:04:57.436662 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:57.436666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:57.436733 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:57.436818 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:57.438000 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.438714 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.441478 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.442473 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.442643 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.447993 (+ 5350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:35013: BlockingRecv error: recv got EOF from 127.1.188.194:35013 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:57.849444 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:57.838752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:35743)
0621 07:04:57.839434 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:57.839454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:57.839581 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:57.839930 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:57.839935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:57.840476 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:57.840661 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.840672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.841648 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.841654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.843212 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.843227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.844087 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.844093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.844270 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.844921 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:57.844942 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.849078 (+ 4136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":425,"threads_started":1}
W20260621 07:04:57.849572 11032 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:57.838801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:35743 (local address 127.1.188.254:46349)
0621 07:04:57.839052 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:57.839072 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:57.839536 (+ 464us) server_negotiation.cc:407] Connection header received
0621 07:04:57.839695 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:57.839700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:57.839768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:57.839863 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:57.840802 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.841508 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.843372 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.843900 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.844009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.849318 (+ 5309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:35743: BlockingRecv error: recv got EOF from 127.1.188.193:35743 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 07:04:57.908910 11028 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.214s user 0.013s sys 0.014s
W20260621 07:04:57.910135 11029 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:56.672900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:49763 (local address 127.1.188.254:46349)
0621 07:04:56.679878 (+ 6978us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:56.679885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:56.681124 (+ 1239us) server_negotiation.cc:407] Connection header received
0621 07:04:56.684074 (+ 2950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:56.684083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:56.684188 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:56.684314 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:56.687663 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.689534 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.692313 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.692967 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.693063 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.909877 (+1216814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:49763: BlockingRecv error: recv got EOF from 127.1.188.195:49763 (error 108)
Metrics: {"server-negotiator.queue_time_us":6840,"thread_start_us":6702,"threads_started":1}
I20260621 07:04:57.910712 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:56.672137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:49763)
0621 07:04:56.680512 (+ 8375us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:56.680561 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:56.683505 (+ 2944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:56.685024 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:56.685043 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:56.686188 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:56.687363 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.687415 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.689913 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.689922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.692154 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:56.692170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:56.693441 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:56.693448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:56.693765 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:56.694952 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:56.694997 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.909660 (+1214663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8002,"thread_start_us":7942,"threads_started":1}
W20260621 07:04:57.911624 11024 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:46349 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.188.254:46349: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:57.920392 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:57.912287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:44137)
0621 07:04:57.912492 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:57.912512 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:57.912616 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:57.912899 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:57.912902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:57.913226 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:57.913403 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.913411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.914340 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.914344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.915294 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.915305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.915970 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.915974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.916089 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.916636 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:57.916652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.920091 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 07:04:57.920501 11029 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:57.912313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:44137 (local address 127.1.188.254:46349)
0621 07:04:57.912572 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:57.912576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:57.912593 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:04:57.912712 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:57.912716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:57.912785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:57.912860 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:57.913516 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.914227 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.915448 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.915833 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.915921 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.920274 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:44137: BlockingRecv error: recv got EOF from 127.1.188.195:44137 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:57 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:57.928974 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:57.921083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:47247)
0621 07:04:57.921240 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:57.921267 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:57.921343 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:57.921596 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:57.921598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:57.921866 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:57.922004 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.922011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.922970 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.922975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.924341 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:57.924353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.924968 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.924973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:57.925110 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.925609 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:57.925624 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:57.928781 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 07:04:57.929260 11024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:04:57.929417 11029 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:57.921182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:47247 (local address 127.1.188.254:46349)
0621 07:04:57.921331 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:57.921335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:57.921350 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:04:57.921415 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:57.921418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:57.921483 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:57.921556 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:57.922112 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.922813 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.924441 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:57.924850 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:57.924941 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:57.928933 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:47247: BlockingRecv error: recv got EOF from 127.1.188.195:47247 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:58.460356 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:58.448871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:45973)
0621 07:04:58.449822 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:58.449848 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:58.450039 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:58.450433 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:58.450438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:58.450974 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:58.451158 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.451167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.452358 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.452365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.453408 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.453420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.454188 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.454194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.454397 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.455080 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:58.455113 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:58.459803 (+ 4690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":695,"threads_started":1}
W20260621 07:04:58.460356 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:58.448900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:45973 (local address 127.1.188.254:46349)
0621 07:04:58.449483 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:58.449489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:58.449945 (+ 456us) server_negotiation.cc:407] Connection header received
0621 07:04:58.450175 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:58.450181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:58.450268 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:58.450386 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:58.451318 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.452208 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.453553 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.454007 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.454151 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.460042 (+ 5891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:45973: BlockingRecv error: recv got EOF from 127.1.188.194:45973 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":322,"threads_started":1}
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:58.861707 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:58.851094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:51459)
0621 07:04:58.851806 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:58.851825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:58.852002 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:58.852514 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:58.852520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:58.853147 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:58.853422 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.853438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.854677 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.854682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.855844 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.855859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.856752 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.856763 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.856973 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.857669 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:58.857690 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:58.861417 (+ 3727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":435,"threads_started":1}
W20260621 07:04:58.861856 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:58.851139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:51459 (local address 127.1.188.254:46349)
0621 07:04:58.851357 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:58.851363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:58.851949 (+ 586us) server_negotiation.cc:407] Connection header received
0621 07:04:58.852192 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:58.852200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:58.852315 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:58.852447 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:58.853592 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.854546 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.856027 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.856496 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.856662 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.861633 (+ 4971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:51459: BlockingRecv error: recv got EOF from 127.1.188.193:51459 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:58 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:58.940811 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:58.930034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:46607)
0621 07:04:58.930745 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:58.930764 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:58.930918 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:58.931268 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:58.931272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:58.931820 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:58.931989 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.931999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.933166 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.933171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.934231 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:58.934242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.935065 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.935070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:58.935203 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.935984 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:58.936004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:58.940539 (+ 4535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":419,"threads_started":1}
W20260621 07:04:58.941012 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:58.930085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:46607 (local address 127.1.188.254:46349)
0621 07:04:58.930319 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:58.930323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:58.930879 (+ 556us) server_negotiation.cc:407] Connection header received
0621 07:04:58.931032 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:58.931036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:58.931122 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:58.931214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:58.932140 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.933011 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.934383 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:58.934870 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:58.934987 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:58.940743 (+ 5756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:46607: BlockingRecv error: recv got EOF from 127.1.188.195:46607 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:59.473196 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:59.461602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:36677)
0621 07:04:59.462352 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:59.462403 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:59.462605 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:59.463474 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:59.463478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:59.463969 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:59.464249 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.464264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.465515 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.465520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.466918 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.466935 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.467812 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.467817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.467960 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.468632 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:59.468650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:59.472791 (+ 4141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":371,"threads_started":1}
W20260621 07:04:59.473351 11039 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:59.461651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:36677 (local address 127.1.188.254:46349)
0621 07:04:59.463097 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:59.463102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:59.463116 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:04:59.463162 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:59.463166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:59.463241 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:59.463333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:59.464412 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.465342 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.467076 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.467654 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.467758 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.473027 (+ 5269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:36677: BlockingRecv error: recv got EOF from 127.1.188.194:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":1338,"thread_start_us":1227,"threads_started":1}
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:59.873258 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:59.862808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:37093)
0621 07:04:59.863473 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:59.863490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:59.863685 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:59.864062 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:59.864067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:59.864525 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:59.864734 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.864743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.865846 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.865854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.867450 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.867466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.868209 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.868214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.868348 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.869081 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:59.869100 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:59.872987 (+ 3887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
W20260621 07:04:59.873514 11039 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:59.863005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:37093 (local address 127.1.188.254:46349)
0621 07:04:59.863283 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:59.863289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:59.863591 (+ 302us) server_negotiation.cc:407] Connection header received
0621 07:04:59.863826 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:59.863835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:59.863908 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:59.864019 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:59.864890 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.865712 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.867623 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.868083 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.868183 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.873229 (+ 5046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:37093: BlockingRecv error: recv got EOF from 127.1.188.193:37093 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:04:59 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:04:59.952625 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:04:59.942165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:37015)
0621 07:04:59.942811 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0621 07:04:59.942829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:04:59.942970 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:04:59.943363 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:04:59.943367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:04:59.943856 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:04:59.944064 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.944073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.945278 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.945283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.946325 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:04:59.946338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.947092 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.947097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:04:59.947251 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.948119 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:04:59.948138 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:04:59.952343 (+ 4205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":375,"threads_started":1}
W20260621 07:04:59.952782 11039 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:04:59.942268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:37015 (local address 127.1.188.254:46349)
0621 07:04:59.942477 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 07:04:59.942482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:04:59.942909 (+ 427us) server_negotiation.cc:407] Connection header received
0621 07:04:59.943118 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:04:59.943123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:04:59.943219 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:04:59.943326 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:04:59.944240 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.945112 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.946470 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:04:59.946927 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:04:59.947025 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:04:59.952522 (+ 5497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:37015: BlockingRecv error: recv got EOF from 127.1.188.195:37015 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:00.483901 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:00.474333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:49073)
0621 07:05:00.474925 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:00.474954 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:00.475127 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:00.475595 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:00.475601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:00.476105 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:00.476297 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.476306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.477421 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.477425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.478361 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.478372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.479048 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.479052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.479177 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.479846 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:00.479865 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:00.483530 (+ 3665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":328,"threads_started":1}
W20260621 07:05:00.484053 11043 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:00.474408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:49073 (local address 127.1.188.254:46349)
0621 07:05:00.474925 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:00.474933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:00.475072 (+ 139us) server_negotiation.cc:407] Connection header received
0621 07:05:00.475253 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:00.475261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:00.475387 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:00.475527 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:00.476482 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.477294 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.478483 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.478922 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.479016 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.483768 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:49073: BlockingRecv error: recv got EOF from 127.1.188.194:49073 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":434,"threads_started":1}
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:00.884419 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:00.874590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:49809)
0621 07:05:00.875192 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:00.875210 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:00.875333 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:00.875663 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:00.875666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:00.876117 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:00.876309 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.876318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.877450 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.877454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.878555 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.878567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.879250 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.879254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.879396 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.880113 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:00.880132 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:00.884133 (+ 4001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
W20260621 07:05:00.884645 11043 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:00.874673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:49809 (local address 127.1.188.254:46349)
0621 07:05:00.874920 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:00.874929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:00.875298 (+ 369us) server_negotiation.cc:407] Connection header received
0621 07:05:00.875457 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:00.875461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:00.875536 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:00.875622 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:00.876458 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.877289 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.878704 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.879122 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.879217 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.884362 (+ 5145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:49809: BlockingRecv error: recv got EOF from 127.1.188.193:49809 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:00 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:00.965029 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:00.954041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:56557)
0621 07:05:00.954657 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:00.954675 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:00.954897 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:00.955287 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:00.955291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:00.955760 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:00.955958 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.955967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.957215 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.957223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.958402 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:00.958418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.959308 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.959313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:00.959575 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.960329 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:00.960348 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:00.964655 (+ 4307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":341,"threads_started":1}
W20260621 07:05:00.965461 11043 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:00.954071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:56557 (local address 127.1.188.254:46349)
0621 07:05:00.954418 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:00.954427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:00.954785 (+ 358us) server_negotiation.cc:407] Connection header received
0621 07:05:00.955040 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:00.955046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:00.955142 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:00.955256 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:00.956122 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.957064 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.958599 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:00.959118 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:00.959283 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:00.964961 (+ 5678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:56557: BlockingRecv error: recv got EOF from 127.1.188.195:56557 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:01.495138 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:01.484971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:58903)
0621 07:05:01.485544 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:01.485568 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:01.485689 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:01.486147 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:01.486150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:01.486589 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:01.486775 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.486784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.487823 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.487827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.488931 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.488944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.489755 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.489759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.489888 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.490677 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:01.490697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:01.494752 (+ 4055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":293,"threads_started":1}
W20260621 07:05:01.495432 11047 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:01.485032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:58903 (local address 127.1.188.254:46349)
0621 07:05:01.485852 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:01.485856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:01.485872 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:05:01.485919 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:01.485923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:01.486003 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:01.486098 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:01.486906 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.487713 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.489063 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.489630 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.489759 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.495092 (+ 5333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:58903: BlockingRecv error: recv got EOF from 127.1.188.194:58903 (error 108)
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":602,"threads_started":1}
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:01.896111 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:01.885568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:45875)
0621 07:05:01.886154 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:01.886170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:01.886352 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:01.886725 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:01.886729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:01.887347 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:01.887589 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.887602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.889196 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.889200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.890323 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.890336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.891101 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.891105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.891228 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.891925 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:01.891945 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:01.895842 (+ 3897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":313,"threads_started":1}
W20260621 07:05:01.896359 11047 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:01.885643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45875 (local address 127.1.188.254:46349)
0621 07:05:01.885890 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:01.885900 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:01.886269 (+ 369us) server_negotiation.cc:407] Connection header received
0621 07:05:01.886508 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:01.886512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:01.886582 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:01.886690 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:01.887747 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.889039 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.890450 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.890941 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.891033 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.896065 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:45875: BlockingRecv error: recv got EOF from 127.1.188.193:45875 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:01 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:01.976648 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:01.966202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:60581)
0621 07:05:01.966882 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:01.966900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:01.967039 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:01.967390 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:01.967460 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:01.967945 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:01.968135 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.968143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.969217 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.969221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.970303 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:01.970314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.971002 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.971006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:01.971166 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.971960 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:01.971986 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:01.976359 (+ 4373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":411,"threads_started":1}
W20260621 07:05:01.976881 11047 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:01.966270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:60581 (local address 127.1.188.254:46349)
0621 07:05:01.966494 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:01.966499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:01.966995 (+ 496us) server_negotiation.cc:407] Connection header received
0621 07:05:01.967147 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:01.967151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:01.967243 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:01.967329 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:01.968269 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.969106 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.970430 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:01.970876 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:01.970969 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:01.976579 (+ 5610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:60581: BlockingRecv error: recv got EOF from 127.1.188.195:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:02.512832 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:02.496280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:55025)
0621 07:05:02.496903 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:02.496928 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:02.497056 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:02.497560 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:02.497564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:02.498050 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:02.498245 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.498254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.499319 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.499323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.500460 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.500475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.501205 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.501212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.501412 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.508432 (+ 7020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:02.508454 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:02.512433 (+ 3979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":352,"threads_started":1}
W20260621 07:05:02.513033 11051 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:02.496607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55025 (local address 127.1.188.254:46349)
0621 07:05:02.497224 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:02.497229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:02.497244 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:05:02.497298 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:02.497303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:02.497377 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:02.497474 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:02.498385 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.499179 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.500590 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.501008 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.501101 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.512708 (+ 11607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:55025: BlockingRecv error: recv got EOF from 127.1.188.194:55025 (error 108)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":377,"threads_started":1}
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:02.907896 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:02.897231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:52207)
0621 07:05:02.897824 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:02.897842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:02.898031 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:02.898406 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:02.898412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:02.898927 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:02.899133 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.899142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.900653 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.900659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.901984 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.901999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.902744 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.902750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.902935 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.903721 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:02.903740 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:02.907530 (+ 3790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":352,"threads_started":1}
W20260621 07:05:02.908021 11051 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:02.897328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:52207 (local address 127.1.188.254:46349)
0621 07:05:02.897581 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:02.897590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:02.897941 (+ 351us) server_negotiation.cc:407] Connection header received
0621 07:05:02.898173 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:02.898183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:02.898253 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:02.898361 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:02.899286 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.900523 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.902130 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.902584 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.902726 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.907763 (+ 5037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:52207: BlockingRecv error: recv got EOF from 127.1.188.193:52207 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:02 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:02.988219 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:02.977860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:53577)
0621 07:05:02.978531 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:02.978550 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:02.978697 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:02.979040 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:02.979044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:02.979590 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:02.979805 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.979820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.980928 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.980932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.982055 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:02.982068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.982812 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.982818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:02.983044 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.983921 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:02.983949 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:02.987967 (+ 4018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
W20260621 07:05:02.988567 11051 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:02.977911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:53577 (local address 127.1.188.254:46349)
0621 07:05:02.978112 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:02.978117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:02.978634 (+ 517us) server_negotiation.cc:407] Connection header received
0621 07:05:02.978806 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:02.978810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:02.978899 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:02.978986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:02.979957 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.980785 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.982199 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:02.982675 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:02.982778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:02.988200 (+ 5422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:53577: BlockingRecv error: recv got EOF from 127.1.188.195:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:03.525063 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:03.514121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:54381)
0621 07:05:03.514766 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:03.514790 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:03.514951 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:03.515584 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:03.515588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:03.516232 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:03.516480 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.516492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.517744 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.517749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.518926 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.518940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.519690 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.519695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.519830 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.520579 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:03.520601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:03.524781 (+ 4180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":377,"threads_started":1}
W20260621 07:05:03.525522 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:03.514208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:54381 (local address 127.1.188.254:46349)
0621 07:05:03.515171 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:03.515177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:03.515195 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:05:03.515274 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:03.515278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:03.515349 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:03.515513 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:03.516641 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.517580 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.519062 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.519538 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.519636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.525183 (+ 5547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:54381: BlockingRecv error: recv got EOF from 127.1.188.194:54381 (error 108)
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":743,"threads_started":1}
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:03.919746 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:03.909253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:42749)
0621 07:05:03.910002 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:03.910019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:03.910196 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:03.910556 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:03.910561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:03.911046 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:03.911250 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.911259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.912371 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.912377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.913381 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.913400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.914111 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.914117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.914291 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.914991 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:03.915012 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:03.919237 (+ 4225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":329,"threads_started":1}
W20260621 07:05:03.919785 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:03.909271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:42749 (local address 127.1.188.254:46349)
0621 07:05:03.909455 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:03.909459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:03.910110 (+ 651us) server_negotiation.cc:407] Connection header received
0621 07:05:03.910327 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:03.910333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:03.910419 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:03.910524 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:03.911393 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.912227 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.913536 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.913954 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.914080 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.919505 (+ 5425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:42749: BlockingRecv error: recv got EOF from 127.1.188.193:42749 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:03 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:03.999929 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:03.989394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:46435)
0621 07:05:03.990030 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:03.990046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:03.990186 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:03.990626 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:03.990630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:03.991178 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:03.991454 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.991481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.992580 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.992584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.993535 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:03.993545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.994294 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.994299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:03.994544 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.995325 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:03.995374 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:03.999569 (+ 4195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":357,"threads_started":1}
W20260621 07:05:04.000068 11055 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:03.989458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:46435 (local address 127.1.188.254:46349)
0621 07:05:03.989647 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:03.989652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:03.990137 (+ 485us) server_negotiation.cc:407] Connection header received
0621 07:05:03.990302 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:03.990368 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:03.990478 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:03.990569 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:03.991616 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.992458 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.993667 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:03.994087 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:03.994257 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:03.999787 (+ 5530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:46435: BlockingRecv error: recv got EOF from 127.1.188.195:46435 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 07:05:04 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:04 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:04.536784 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:04.526186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:39775)
0621 07:05:04.526912 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:04.526937 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:04.527100 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:04.527588 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:04.527593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:04.528077 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:04.528338 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:04.528352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.529795 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.529801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:04.530976 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:04.530989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.531835 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.531839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:04.531977 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:04.532689 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:04.532709 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:04.536477 (+ 3768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":341,"threads_started":1}
W20260621 07:05:04.537014 11059 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:04.526331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:39775 (local address 127.1.188.254:46349)
0621 07:05:04.526948 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:04.526953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:04.527033 (+ 80us) server_negotiation.cc:407] Connection header received
0621 07:05:04.527268 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:04.527274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:04.527385 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:04.527537 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:04.528538 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.529609 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.531145 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.531682 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.531800 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:04.536725 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:39775: BlockingRecv error: recv got EOF from 127.1.188.194:39775 (error 108)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":381,"threads_started":1}
Jun 21 07:05:04 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:04 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:04.932015 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:04.921017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:43995)
0621 07:05:04.921653 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:04.921670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:04.921805 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:04.922203 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:04.922208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:04.922719 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:04.923088 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:04.923109 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.924378 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.924382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:04.925516 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:04.925530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.926312 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.926317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:04.926467 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:04.927249 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:04.927268 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:04.931736 (+ 4468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
W20260621 07:05:04.932183 11059 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:04.921085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:43995 (local address 127.1.188.254:46349)
0621 07:05:04.921334 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:04.921344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:04.921763 (+ 419us) server_negotiation.cc:407] Connection header received
0621 07:05:04.921916 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:04.921920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:04.921997 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:04.922099 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:04.923250 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.924208 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.925662 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:04.926109 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:04.926214 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:04.931942 (+ 5728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:43995: BlockingRecv error: recv got EOF from 127.1.188.193:43995 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:05.011914 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:05.001319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:55559)
0621 07:05:05.002321 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:05.002339 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:05.002534 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:05.003006 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:05.003010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:05.003564 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:05.003759 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.003769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.004863 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.004872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.005882 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.005892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.006626 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.006633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.006820 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.007614 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:05.007632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:05.011634 (+ 4002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":469,"threads_started":1}
W20260621 07:05:05.012192 11059 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:05.001541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:55559 (local address 127.1.188.254:46349)
0621 07:05:05.001784 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:05.001793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:05.002437 (+ 644us) server_negotiation.cc:407] Connection header received
0621 07:05:05.002732 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:05.002738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:05.002854 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:05.002981 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:05.003906 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.004719 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.006028 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.006466 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.006606 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.011906 (+ 5300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:55559: BlockingRecv error: recv got EOF from 127.1.188.195:55559 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:05.548135 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:05.537959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:36781)
0621 07:05:05.538567 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:05.538590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:05.538712 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:05.539212 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:05.539215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:05.539760 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:05.539959 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.539969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.541157 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.541160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.542135 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.542146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.542872 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.542877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.542995 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.543855 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:05.543882 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:05.547736 (+ 3854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
W20260621 07:05:05.548251 11063 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:05.538064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:36781 (local address 127.1.188.254:46349)
0621 07:05:05.538900 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:05.538905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:05.538921 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:05:05.538983 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:05.538987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:05.539055 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:05.539159 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:05.540085 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.541030 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.542261 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.542718 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.542811 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.547959 (+ 5148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:36781: BlockingRecv error: recv got EOF from 127.1.188.194:36781 (error 108)
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":603,"threads_started":1}
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:05 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:05.943516 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:05.933270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:47029)
0621 07:05:05.933957 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:05.933975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:05.934106 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:05.934437 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:05.934441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:05.934919 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:05.935108 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.935118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.936605 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.936612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.937897 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:05.937914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.938585 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.938590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:05.938725 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.939365 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:05.939384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:05.943191 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":384,"threads_started":1}
W20260621 07:05:05.943655 11063 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:05.933355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:47029 (local address 127.1.188.254:46349)
0621 07:05:05.933554 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:05.933559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:05.934055 (+ 496us) server_negotiation.cc:407] Connection header received
0621 07:05:05.934224 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:05.934228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:05.934300 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:05.934387 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:05.935253 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.936406 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.938039 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:05.938438 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:05.938542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:05.943368 (+ 4826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:47029: BlockingRecv error: recv got EOF from 127.1.188.193:47029 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:06.023195 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:06.013071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:34903)
0621 07:05:06.013664 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:06.013682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:06.013853 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:06.014227 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:06.014232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:06.014729 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:06.014916 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.014925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.016098 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.016105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.017116 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.017127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.017885 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.017891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.018075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.018866 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:06.018884 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:06.022896 (+ 4012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":313,"threads_started":1}
W20260621 07:05:06.023475 11063 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:06.013115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:34903 (local address 127.1.188.254:46349)
0621 07:05:06.013320 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:06.013329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:06.013774 (+ 445us) server_negotiation.cc:407] Connection header received
0621 07:05:06.013985 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:06.013991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:06.014089 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:06.014195 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:06.015066 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.015954 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.017259 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.017697 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.017834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.023128 (+ 5294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:34903: BlockingRecv error: recv got EOF from 127.1.188.195:34903 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:06.559530 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:06.549272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:36443)
0621 07:05:06.549914 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:06.549938 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:06.550060 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:06.550447 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:06.550464 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:06.550947 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:06.551132 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.551141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.552315 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.552319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.553375 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.553388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.554088 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.554092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.554277 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.554951 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:06.554972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:06.559210 (+ 4238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":330,"threads_started":1}
W20260621 07:05:06.559742 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:06.549609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:36443 (local address 127.1.188.254:46349)
0621 07:05:06.550144 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:06.550149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:06.550165 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:05:06.550214 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:06.550218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:06.550298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:06.550399 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:06.551278 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.552170 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.553492 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.553953 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.554051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.559458 (+ 5407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:36443: BlockingRecv error: recv got EOF from 127.1.188.194:36443 (error 108)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":298,"threads_started":1}
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:06 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:06.955613 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:06.944888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:52827)
0621 07:05:06.945470 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:06.945489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:06.945629 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:06.945991 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:06.945994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:06.946436 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:06.946614 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.946622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.948206 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.948211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.949574 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:06.949587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.950264 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.950268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:06.950379 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.951070 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:06.951089 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:06.955213 (+ 4124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":331,"threads_started":1}
W20260621 07:05:06.955762 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:06.944996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:52827 (local address 127.1.188.254:46349)
0621 07:05:06.945281 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:06.945289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:06.945584 (+ 295us) server_negotiation.cc:407] Connection header received
0621 07:05:06.945751 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:06.945755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:06.945843 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:06.945935 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:06.946863 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.948041 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.949707 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:06.950135 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:06.950233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:06.955507 (+ 5274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:52827: BlockingRecv error: recv got EOF from 127.1.188.193:52827 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:07.034955 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:07.024557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:34199)
0621 07:05:07.025190 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:07.025208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:07.025383 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:07.025763 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:07.025768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:07.026287 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:07.026453 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.026462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.027648 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.027655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.028690 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.028701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.029444 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.029449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.029643 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.030298 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:07.030315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:07.034667 (+ 4352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":325,"threads_started":1}
W20260621 07:05:07.035118 11067 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:07.024651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:34199 (local address 127.1.188.254:46349)
0621 07:05:07.024924 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:07.024933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:07.025301 (+ 368us) server_negotiation.cc:407] Connection header received
0621 07:05:07.025518 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:07.025524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:07.025623 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:07.025729 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:07.026608 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.027514 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.028843 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.029290 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.029430 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.034856 (+ 5426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:34199: BlockingRecv error: recv got EOF from 127.1.188.195:34199 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
W20260621 07:05:07.574213 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:07.560663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55289 (local address 127.1.188.254:46349)
0621 07:05:07.561181 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:07.561187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:07.561282 (+ 95us) server_negotiation.cc:407] Connection header received
0621 07:05:07.561453 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:07.561457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:07.561542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:07.561645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:07.562471 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.563539 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.565290 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.565719 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.565813 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.573963 (+ 8150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:55289: BlockingRecv error: recv got EOF from 127.1.188.194:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
I20260621 07:05:07.574331 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:07.560557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:55289)
0621 07:05:07.561177 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:07.561200 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:07.561345 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:07.561690 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:07.561693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:07.562160 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:07.562330 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.562339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.563708 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.563714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.565162 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.565175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.565842 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.565846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.566023 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.569879 (+ 3856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:07.569900 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:07.573756 (+ 3856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":344,"threads_started":1}
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:07 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:07.967198 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:07.956744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:48047)
0621 07:05:07.957376 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:07.957393 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:07.957574 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:07.957943 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:07.957948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:07.958474 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:07.958685 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.958694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.959893 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.959900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.960908 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:07.960920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.961678 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.961684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:07.961877 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.962690 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:07.962710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:07.966795 (+ 4085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":349,"threads_started":1}
W20260621 07:05:07.967280 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:07.956951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:48047 (local address 127.1.188.254:46349)
0621 07:05:07.957206 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:07.957213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:07.957501 (+ 288us) server_negotiation.cc:407] Connection header received
0621 07:05:07.957712 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:07.957724 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:07.957796 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:07.957905 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:07.958853 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.959758 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.961056 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:07.961523 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:07.961663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:07.967044 (+ 5381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:48047: BlockingRecv error: recv got EOF from 127.1.188.193:48047 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:08.046182 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:08.036249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:57015)
0621 07:05:08.036836 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:08.036853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:08.037033 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:08.037418 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:08.037423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:08.037894 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:08.038084 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.038094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.039215 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.039223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.040297 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.040309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.041055 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.041061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.041240 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.041984 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:08.042003 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:08.045747 (+ 3744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":331,"threads_started":1}
W20260621 07:05:08.046289 11071 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:08.036280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:57015 (local address 127.1.188.254:46349)
0621 07:05:08.036499 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:08.036507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:08.036950 (+ 443us) server_negotiation.cc:407] Connection header received
0621 07:05:08.037165 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:08.037171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:08.037272 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:08.037380 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:08.038239 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.039066 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.040443 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.040899 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.041048 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.045984 (+ 4936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:57015: BlockingRecv error: recv got EOF from 127.1.188.195:57015 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:08.585180 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:08.575536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:47311)
0621 07:05:08.576130 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:08.576154 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:08.576289 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:08.576644 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:08.576647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:08.577119 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:08.577335 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.577343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.578384 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.578388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.579368 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.579380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.580137 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.580142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.580329 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.581095 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:08.581115 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:08.584906 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":313,"threads_started":1}
W20260621 07:05:08.585543 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:08.575574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:47311 (local address 127.1.188.254:46349)
0621 07:05:08.576100 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:08.576105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:08.576234 (+ 129us) server_negotiation.cc:407] Connection header received
0621 07:05:08.576397 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:08.576401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:08.576496 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:08.576591 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:08.577461 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.578269 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.579523 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.579985 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.580081 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.585192 (+ 5111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:47311: BlockingRecv error: recv got EOF from 127.1.188.194:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:08 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:08.979007 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:08.968535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:43615)
0621 07:05:08.969145 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:08.969163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:08.969356 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:08.969780 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:08.969787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:08.970286 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:08.970499 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.970510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.971751 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.971757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.973059 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:08.973077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.974001 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.974005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:08.974242 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.974997 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:08.975018 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:08.978717 (+ 3699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":333,"threads_started":1}
W20260621 07:05:08.979148 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:08.968609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:43615 (local address 127.1.188.254:46349)
0621 07:05:08.968910 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:08.968918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:08.969270 (+ 352us) server_negotiation.cc:407] Connection header received
0621 07:05:08.969504 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:08.969510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:08.969618 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:08.969733 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:08.970664 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.971548 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.973234 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:08.973863 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:08.973958 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:08.978925 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:43615: BlockingRecv error: recv got EOF from 127.1.188.193:43615 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:09.057827 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:09.047701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:49137)
0621 07:05:09.048313 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:09.048332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:09.048492 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:09.048884 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:09.048888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:09.049512 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:09.049724 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.049733 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.050787 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.050791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.051806 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.051817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.052615 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.052619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.052760 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.053490 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:09.053510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:09.057429 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":307,"threads_started":1}
W20260621 07:05:09.057930 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:09.047771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:49137 (local address 127.1.188.254:46349)
0621 07:05:09.048055 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:09.048065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:09.048449 (+ 384us) server_negotiation.cc:407] Connection header received
0621 07:05:09.048623 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:09.048627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:09.048722 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:09.048836 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:09.049863 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.050675 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.051982 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.052475 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.052581 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.057638 (+ 5057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:49137: BlockingRecv error: recv got EOF from 127.1.188.195:49137 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:09.596712 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:09.586577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:59529)
0621 07:05:09.587191 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:09.587215 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:09.587368 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:09.587787 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:09.587790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:09.588208 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:09.588411 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.588420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.589759 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.589765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.590942 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.590957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.591794 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.591800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.592004 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.592673 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:09.592692 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:09.596408 (+ 3716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
W20260621 07:05:09.596925 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:09.586597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:59529 (local address 127.1.188.254:46349)
0621 07:05:09.587176 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:09.587181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:09.587297 (+ 116us) server_negotiation.cc:407] Connection header received
0621 07:05:09.587510 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:09.587516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:09.587618 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:09.587751 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:09.588561 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.589611 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.591102 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.591652 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.591782 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.596646 (+ 4864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:59529: BlockingRecv error: recv got EOF from 127.1.188.194:59529 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":333,"threads_started":1}
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:09 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:09.990121 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:09.980109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:45309)
0621 07:05:09.980682 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:09.980699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:09.980891 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:09.981244 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:09.981249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:09.981830 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:09.982011 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.982019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.983106 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.983113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.984165 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:09.984177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.984955 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.984959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:09.985090 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.985798 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:09.985818 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:09.989718 (+ 3900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
W20260621 07:05:09.990242 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:09.980186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45309 (local address 127.1.188.254:46349)
0621 07:05:09.980440 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:09.980448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:09.980794 (+ 346us) server_negotiation.cc:407] Connection header received
0621 07:05:09.981025 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:09.981033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:09.981102 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:09.981209 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:09.982166 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.982971 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.984375 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:09.984832 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:09.984934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:09.989961 (+ 5027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:45309: BlockingRecv error: recv got EOF from 127.1.188.193:45309 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:10.070459 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:10.059150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:47771)
0621 07:05:10.059911 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:10.059934 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:10.060113 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:10.060645 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:10.060652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:10.061187 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:10.061415 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.061427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.062604 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.062611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.063764 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.063778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.064629 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.064636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.064857 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:10.065674 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:10.065694 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:10.070170 (+ 4476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":417,"threads_started":1}
W20260621 07:05:10.070719 11078 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:10.059154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:47771 (local address 127.1.188.254:46349)
0621 07:05:10.059422 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:10.059432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:10.060060 (+ 628us) server_negotiation.cc:407] Connection header received
0621 07:05:10.060269 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:10.060275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:10.060442 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:10.060581 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:10.061591 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.062434 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.063925 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.064423 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.064583 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:10.070411 (+ 5828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:47771: BlockingRecv error: recv got EOF from 127.1.188.195:47771 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:10.608492 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:10.597982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:50761)
0621 07:05:10.598820 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:10.598846 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:10.598988 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:10.599341 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:10.599344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:10.599815 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:10.599996 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.600005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.601080 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.601085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.602088 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.602103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.602897 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.602902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.603046 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:10.603750 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:10.603771 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:10.608209 (+ 4438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":551,"threads_started":1}
W20260621 07:05:10.608767 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:10.598155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:50761 (local address 127.1.188.254:46349)
0621 07:05:10.598864 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:10.598869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:10.598912 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:05:10.599107 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:10.599111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:10.599188 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:10.599285 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:10.600135 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.600929 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.602274 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.602705 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.602810 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:10.608480 (+ 5670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:50761: BlockingRecv error: recv got EOF from 127.1.188.194:50761 (error 108)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":377,"threads_started":1}
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:10 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.193@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:11.002316 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:10.991800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.193:36023)
0621 07:05:10.992342 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:10.992360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:10.992501 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:10.992817 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:10.992821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:10.993391 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:10.993687 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.993705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.995192 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.995196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.996200 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:10.996211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.996898 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.996902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:10.997046 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:10.997740 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:10.997759 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:11.001939 (+ 4180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260621 07:05:11.002368 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:10.991853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:36023 (local address 127.1.188.254:46349)
0621 07:05:10.992113 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:10.992122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:10.992451 (+ 329us) server_negotiation.cc:407] Connection header received
0621 07:05:10.992612 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:10.992617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:10.992691 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:10.992776 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:10.993843 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.995048 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.996330 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:10.996765 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:10.996874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.002162 (+ 5288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.193:36023: BlockingRecv error: recv got EOF from 127.1.188.193:36023 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 07:05:11 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:11 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.195@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:11.081923 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:11.071758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.195:48787)
0621 07:05:11.072332 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:11.072351 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:11.072501 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:11.072827 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:11.072830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:11.073274 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:11.073479 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:11.073514 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.074740 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.074746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:11.075880 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:11.075896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.076669 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.076674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:11.076815 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.077557 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:11.077576 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:11.081657 (+ 4081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":325,"threads_started":1}
W20260621 07:05:11.082228 11083 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:11.071784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:48787 (local address 127.1.188.254:46349)
0621 07:05:11.072101 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:11.072106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:11.072445 (+ 339us) server_negotiation.cc:407] Connection header received
0621 07:05:11.072607 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:11.072611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:11.072699 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:11.072788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:11.073650 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.074506 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.076047 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.076508 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.076605 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.081888 (+ 5283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.195:48787: BlockingRecv error: recv got EOF from 127.1.188.195:48787 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
Jun 21 07:05:11 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
Jun 21 07:05:11 dist-test-slave-hlq1 krb5kdc[10022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.188.194@KRBTEST.COM for oryx/127.1.188.254@KRBTEST.COM, Server not found in Kerberos database
I20260621 07:05:11.620478 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:11.610057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:46349 (local address 127.1.188.194:41741)
0621 07:05:11.610691 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:11.610716 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:11.610848 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:11.611239 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:11.611243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:11.611731 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:11.611960 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:11.611972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.613091 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.613096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:11.614090 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:11.614102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.615120 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.615126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:11.615327 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.616143 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:11.616162 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:11.620077 (+ 3915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.188.254:46349: Server oryx/127.1.188.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":325,"threads_started":1}
W20260621 07:05:11.620599 11087 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:11.610150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:41741 (local address 127.1.188.254:46349)
0621 07:05:11.610753 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:11.610758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:11.610779 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:05:11.610971 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:11.610976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:11.611051 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:11.611172 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:11.612109 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.612919 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.614261 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.614966 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.615111 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.620302 (+ 5191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.188.194:41741: BlockingRecv error: recv got EOF from 127.1.188.194:41741 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":373,"threads_started":1}
I20260621 07:05:11.663705 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10610
I20260621 07:05:11.678285 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:11.665213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.1.188.193:46839)
0621 07:05:11.666066 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:11.666076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:11.666114 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:05:11.666202 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:11.666210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:11.666523 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:11.666744 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:11.668554 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.669938 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.671243 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.671931 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.672039 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.674900 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:11.674910 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:11.674937 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:11.674991 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:11.677682 (+ 2691us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 07:05:11.678036 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57324: Request ticket server kudu/127.1.188.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":431,"threads_started":1}
W20260621 07:05:11.678424 11089 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57324: Request ticket server kudu/127.1.188.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 07:05:11.678488 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10610: Not authorized: Client connection negotiation failed: client connection to 127.1.188.193:46839: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 07:05:11.678727 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10610
I20260621 07:05:11.698354 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10762
I20260621 07:05:11.716446 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:11.699850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.1.188.194:43635)
0621 07:05:11.701060 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:11.701072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:11.701116 (+ 44us) server_negotiation.cc:407] Connection header received
0621 07:05:11.701220 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:11.701232 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:11.701646 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:11.701888 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:11.704636 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.706469 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.708237 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.709073 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.709224 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.712634 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:11.712645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:11.712664 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:11.712716 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:11.715789 (+ 3073us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 07:05:11.716149 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56108: Request ticket server kudu/127.1.188.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":644,"threads_started":1}
W20260621 07:05:11.716590 11090 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56108: Request ticket server kudu/127.1.188.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 07:05:11.716640 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10762: Not authorized: Client connection negotiation failed: client connection to 127.1.188.194:43635: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 07:05:11.716804 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10762
I20260621 07:05:11.735523 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10895
I20260621 07:05:11.749862 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:11.737035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.1.188.195:37807)
0621 07:05:11.737727 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:11.737736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:11.737768 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:05:11.737853 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:11.737861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:11.738156 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:11.738406 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:11.740002 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.741157 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.742488 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:11.743139 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:11.743298 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:11.746052 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:11.746062 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:11.746080 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:11.746132 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:11.749046 (+ 2914us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 07:05:11.749458 (+ 412us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43040: Request ticket server kudu/127.1.188.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":328,"threads_started":1}
W20260621 07:05:11.749912 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10895: Not authorized: Client connection negotiation failed: client connection to 127.1.188.195:37807: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.188.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 07:05:11.750000 11091 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43040: Request ticket server kudu/127.1.188.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 07:05:11.750138 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10895
I20260621 07:05:11.769258 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 10541
I20260621 07:05:11.771025 10603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:11.907456 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 10541
2026-06-21T07:05:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21708 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T07:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:05:11Z Disabled control of system clock
I20260621 07:05:12.001641 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39697
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 07:05:12.425326 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:12.425684 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:12.425832 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:12.433699 11096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:05:12.433838 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:12.433909 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:12.434003 11096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:05:12.434091 11096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:05:12.444351 11096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:39697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:12.446164 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:12.448567 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:12.463783 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:12.463820 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:12.463783 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:12.465135 11096 server_base.cc:1034] running on GCE node
I20260621 07:05:12.466487 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:12.468544 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:12.469797 11096 hybrid_clock.cc:648] HybridClock initialized: now 1782025512469745 us; error 48 us; skew 500 ppm
I20260621 07:05:12.470234 11096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:12.473052 11096 webserver.cc:460] Webserver started at http://127.1.188.254:33365/ using document root <none> and password file <none>
I20260621 07:05:12.473929 11096 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:12.474076 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:12.474545 11096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:12.478309 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "121c87df4efc41898041c21aacee2685"
format_stamp: "Formatted at 2026-06-21 07:05:12 on dist-test-slave-hlq1"
server_key: "2341642ba2a2eb998a0797106b3d7d8a"
server_key_iv: "063b2ba8cb72b1709196a78c1aea4278"
server_key_version: "encryptionkey@0"
I20260621 07:05:12.479454 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "121c87df4efc41898041c21aacee2685"
format_stamp: "Formatted at 2026-06-21 07:05:12 on dist-test-slave-hlq1"
server_key: "2341642ba2a2eb998a0797106b3d7d8a"
server_key_iv: "063b2ba8cb72b1709196a78c1aea4278"
server_key_version: "encryptionkey@0"
I20260621 07:05:12.485796 11096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:05:12.489969 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:12.491657 11096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:05:12.491858 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "121c87df4efc41898041c21aacee2685"
format_stamp: "Formatted at 2026-06-21 07:05:12 on dist-test-slave-hlq1"
server_key: "2341642ba2a2eb998a0797106b3d7d8a"
server_key_iv: "063b2ba8cb72b1709196a78c1aea4278"
server_key_version: "encryptionkey@0"
I20260621 07:05:12.492126 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:12.509317 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:12.510301 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:12.510617 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:12.532864 11096 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:39697
I20260621 07:05:12.532893 11161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:39697 every 8 connection(s)
I20260621 07:05:12.534732 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:05:12.539788 11162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:12.540210 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11096
I20260621 07:05:12.540423 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:05:12.541204 1779 external_mini_cluster.cc:1428] Setting key 096b4e018888c1b3a02dbd3a411757a0
I20260621 07:05:12.557200 11162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685: Bootstrap starting.
I20260621 07:05:12.559903 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:12.543666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.1.188.254:39697)
0621 07:05:12.544935 (+ 1269us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:12.544953 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:12.545015 (+ 62us) server_negotiation.cc:407] Connection header received
0621 07:05:12.547741 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:12.547780 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:12.548630 (+ 850us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:12.549220 (+ 590us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:12.549829 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:12.551927 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:12.553452 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:12.554716 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:12.554917 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:12.556372 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:12.556385 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:12.556412 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:12.557308 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:12.557435 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:12.557462 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:12.557589 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:12.557854 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 07:05:12.558304 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":380,"threads_started":1}
I20260621 07:05:12.561987 11162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:12.563556 11162 log.cc:824] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:12.567166 11162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685: No bootstrap required, opened a new log
I20260621 07:05:12.576386 11162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } }
I20260621 07:05:12.576999 11162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:12.577134 11162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c87df4efc41898041c21aacee2685, State: Initialized, Role: FOLLOWER
I20260621 07:05:12.578032 11162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } }
I20260621 07:05:12.578317 11162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:05:12.578441 11162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:05:12.578589 11162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:12.581084 11162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } }
I20260621 07:05:12.581851 11162 leader_election.cc:302] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 121c87df4efc41898041c21aacee2685; no voters:
I20260621 07:05:12.582643 11162 leader_election.cc:288] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:05:12.582768 11167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:05:12.583547 11167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [term 1 LEADER]: Becoming Leader. State: Replica: 121c87df4efc41898041c21aacee2685, State: Running, Role: LEADER
I20260621 07:05:12.584125 11167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } }
I20260621 07:05:12.585095 11162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:05:12.587349 11169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 121c87df4efc41898041c21aacee2685. Latest consensus state: current_term: 1 leader_uuid: "121c87df4efc41898041c21aacee2685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } } }
I20260621 07:05:12.587239 11168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "121c87df4efc41898041c21aacee2685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "121c87df4efc41898041c21aacee2685" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 39697 } } }
I20260621 07:05:12.587693 11169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:12.587839 11168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:12.588732 11172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:05:12.595883 11172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:05:12.606949 11172 catalog_manager.cc:1347] Generated new cluster ID: 740a98844f0d4ac8aa0aa7d0fa6d931d
I20260621 07:05:12.607090 11172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:05:12.617553 11172 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:05:12.619383 11172 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:05:12.634208 11172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 121c87df4efc41898041c21aacee2685: Generated new TSK 0
I20260621 07:05:12.635439 11172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:05:12.648288 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--builtin_ntp_servers=127.1.188.212:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 07:05:13.041000 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:13.041415 11186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:13.041512 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:13.049397 11186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:13.049528 11186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:13.049654 11186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:05:13.059615 11186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:13.061237 11186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:13.063340 11186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:13.078524 11191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:13.078547 11194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:13.078519 11192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:13.079377 11186 server_base.cc:1034] running on GCE node
I20260621 07:05:13.080144 11186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:13.081233 11186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:13.082453 11186 hybrid_clock.cc:648] HybridClock initialized: now 1782025513082397 us; error 48 us; skew 500 ppm
I20260621 07:05:13.082834 11186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:13.090777 11186 webserver.cc:460] Webserver started at http://127.1.188.193:33493/ using document root <none> and password file <none>
I20260621 07:05:13.091813 11186 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:13.091944 11186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:13.092347 11186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:13.096223 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "7080661d8e00425a8bcd055ce49b6420"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "7a57621eae12070218c39b55aa446b40"
server_key_iv: "8ec9a882423d52f31dfe8fdd931c4975"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.097359 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "7080661d8e00425a8bcd055ce49b6420"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "7a57621eae12070218c39b55aa446b40"
server_key_iv: "8ec9a882423d52f31dfe8fdd931c4975"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.103663 11186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:05:13.107601 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.109238 11186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:05:13.109447 11186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "7080661d8e00425a8bcd055ce49b6420"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "7a57621eae12070218c39b55aa446b40"
server_key_iv: "8ec9a882423d52f31dfe8fdd931c4975"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.109699 11186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:13.130211 11186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:13.131320 11186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:13.131727 11186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:13.132875 11186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:13.134599 11186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:13.134719 11186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.134867 11186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:13.134958 11186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.171847 11186 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:40573
I20260621 07:05:13.171890 11312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:40573 every 8 connection(s)
I20260621 07:05:13.173447 11186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:05:13.178437 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11186
I20260621 07:05:13.178668 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:05:13.179590 1779 external_mini_cluster.cc:1428] Setting key 507d483484382d2832e9b17f806e416a
I20260621 07:05:13.183032 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--builtin_ntp_servers=127.1.188.212:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:05:13.189596 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:13.176389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:51779 (local address 127.1.188.254:39697)
0621 07:05:13.176939 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:13.176944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:13.178198 (+ 1254us) server_negotiation.cc:407] Connection header received
0621 07:05:13.180947 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:13.180953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:13.181038 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:13.181169 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:13.182690 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.183633 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.186325 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.186815 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.186918 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:13.188770 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:13.188775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:13.188778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:13.188802 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:13.188846 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:13.188859 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:13.188928 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:13.189203 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:05:13.189292 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260621 07:05:13.191332 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:13.176926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39697 (local address 127.1.188.193:51779)
0621 07:05:13.177980 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:13.178037 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:13.180615 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:13.181396 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:13.181405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:13.181525 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:13.182458 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:13.182483 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.183804 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.183814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:13.186152 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:13.186169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.187064 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.187073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:13.187441 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:13.188366 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:13.188371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:13.188432 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:13.188552 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:13.188956 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:13.188969 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:13.188980 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:05:13.189191 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 07:05:13.189688 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":301,"threads_started":1}
I20260621 07:05:13.193318 11313 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39697
I20260621 07:05:13.193689 11313 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:13.194666 11313 heartbeater.cc:511] Master 127.1.188.254:39697 requested a full tablet report, sending...
I20260621 07:05:13.197582 11127 ts_manager.cc:194] Registered new tserver with Master: 7080661d8e00425a8bcd055ce49b6420 (127.1.188.193:40573)
I20260621 07:05:13.200007 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.193:51779
W20260621 07:05:13.587098 11317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:13.587492 11317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:13.587600 11317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:13.596670 11317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:13.596808 11317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:13.596998 11317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:05:13.607580 11317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:13.609220 11317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:13.611387 11317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:13.629587 11323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:13.629822 11322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:13.630959 11325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:13.631532 11317 server_base.cc:1034] running on GCE node
I20260621 07:05:13.632314 11317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:13.633561 11317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:13.634789 11317 hybrid_clock.cc:648] HybridClock initialized: now 1782025513634732 us; error 50 us; skew 500 ppm
I20260621 07:05:13.635192 11317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:13.638108 11317 webserver.cc:460] Webserver started at http://127.1.188.194:45099/ using document root <none> and password file <none>
I20260621 07:05:13.639011 11317 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:13.639158 11317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:13.639633 11317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:13.643806 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "1498bef193894117b3875b3b046d3040"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "6d571a7bd86cd87a532c4bc84961eff0"
server_key_iv: "5a4b973eab7f9d4e1c502eead3c46fbe"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.645022 11317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "1498bef193894117b3875b3b046d3040"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "6d571a7bd86cd87a532c4bc84961eff0"
server_key_iv: "5a4b973eab7f9d4e1c502eead3c46fbe"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.653436 11317 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260621 07:05:13.657618 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.659355 11317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:05:13.659618 11317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "1498bef193894117b3875b3b046d3040"
format_stamp: "Formatted at 2026-06-21 07:05:13 on dist-test-slave-hlq1"
server_key: "6d571a7bd86cd87a532c4bc84961eff0"
server_key_iv: "5a4b973eab7f9d4e1c502eead3c46fbe"
server_key_version: "encryptionkey@0"
I20260621 07:05:13.659864 11317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:13.680045 11317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:13.681172 11317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:13.681533 11317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:13.682724 11317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:13.684566 11317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:13.684661 11317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.684829 11317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:13.684939 11317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:13.723589 11317 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:38549
I20260621 07:05:13.723608 11443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:38549 every 8 connection(s)
I20260621 07:05:13.725178 11317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:05:13.728379 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11317
I20260621 07:05:13.728572 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:05:13.729146 1779 external_mini_cluster.cc:1428] Setting key 477d3051f246f250790661e2634bc5da
I20260621 07:05:13.732825 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--builtin_ntp_servers=127.1.188.212:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 07:05:13.741779 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:13.728395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55205 (local address 127.1.188.254:39697)
0621 07:05:13.729032 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:13.729041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:13.730582 (+ 1541us) server_negotiation.cc:407] Connection header received
0621 07:05:13.733198 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:13.733202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:13.733278 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:13.733379 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:13.735093 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.735964 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.738507 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.738948 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.739057 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:13.740925 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:13.740930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:13.740933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:13.740955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:13.741003 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:13.741017 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:13.741092 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:13.741367 (+ 275us) server_negotiation.cc:299] Negotiation successful
0621 07:05:13.741459 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":316,"threads_started":1}
I20260621 07:05:13.743386 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:13.729176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39697 (local address 127.1.188.194:55205)
0621 07:05:13.730439 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:13.730487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:13.732825 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:13.733713 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:13.733729 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:13.733891 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:13.734904 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:13.734932 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.736120 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.736128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:13.738317 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:13.738341 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:13.739192 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:13.739201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:13.739595 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:13.740504 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:13.740510 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:13.740577 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:13.740702 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:13.741118 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:13.741131 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:13.741142 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:05:13.741347 (+ 205us) client_negotiation.cc:240] Negotiation successful
0621 07:05:13.741844 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":344,"threads_started":1}
I20260621 07:05:13.745241 11444 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39697
I20260621 07:05:13.745602 11444 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:13.746537 11444 heartbeater.cc:511] Master 127.1.188.254:39697 requested a full tablet report, sending...
I20260621 07:05:13.748379 11127 ts_manager.cc:194] Registered new tserver with Master: 1498bef193894117b3875b3b046d3040 (127.1.188.194:38549)
I20260621 07:05:13.749302 11127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.194:55205
W20260621 07:05:14.127079 11448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:14.127527 11448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:14.127687 11448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:14.135774 11448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:14.135890 11448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:14.136083 11448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:05:14.146092 11448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:14.147946 11448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:14.150242 11448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:14.165520 11453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:14.165539 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:14.165720 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:14.168978 11448 server_base.cc:1034] running on GCE node
I20260621 07:05:14.169716 11448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:14.170888 11448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:14.172147 11448 hybrid_clock.cc:648] HybridClock initialized: now 1782025514172043 us; error 125 us; skew 500 ppm
I20260621 07:05:14.172587 11448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:14.175752 11448 webserver.cc:460] Webserver started at http://127.1.188.195:36169/ using document root <none> and password file <none>
I20260621 07:05:14.176792 11448 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:14.176929 11448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:14.177364 11448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:14.181012 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "f46f9a26f892433bae8c97d09b46a750"
format_stamp: "Formatted at 2026-06-21 07:05:14 on dist-test-slave-hlq1"
server_key: "429893fda6863d07fc712c380936be2a"
server_key_iv: "cec2fa001a70532860d3fc1e22d58919"
server_key_version: "encryptionkey@0"
I20260621 07:05:14.181840 11448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "f46f9a26f892433bae8c97d09b46a750"
format_stamp: "Formatted at 2026-06-21 07:05:14 on dist-test-slave-hlq1"
server_key: "429893fda6863d07fc712c380936be2a"
server_key_iv: "cec2fa001a70532860d3fc1e22d58919"
server_key_version: "encryptionkey@0"
I20260621 07:05:14.187881 11448 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260621 07:05:14.191748 11462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:14.193423 11448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:05:14.193622 11448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "f46f9a26f892433bae8c97d09b46a750"
format_stamp: "Formatted at 2026-06-21 07:05:14 on dist-test-slave-hlq1"
server_key: "429893fda6863d07fc712c380936be2a"
server_key_iv: "cec2fa001a70532860d3fc1e22d58919"
server_key_version: "encryptionkey@0"
I20260621 07:05:14.193837 11448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:14.203645 11313 heartbeater.cc:503] Master 127.1.188.254:39697 was elected leader, sending a full tablet report...
I20260621 07:05:14.216401 11448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:14.217463 11448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:14.217852 11448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:14.218997 11448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:14.220728 11448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:14.220858 11448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:14.221004 11448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:14.221103 11448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:14.258771 11448 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:35589
I20260621 07:05:14.258816 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:35589 every 8 connection(s)
I20260621 07:05:14.260520 11448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:05:14.265978 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11448
I20260621 07:05:14.266188 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:05:14.266798 1779 external_mini_cluster.cc:1428] Setting key 68b2b9d78cac172dd65b0612231c9400
I20260621 07:05:14.276158 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:14.263718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:60345 (local address 127.1.188.254:39697)
0621 07:05:14.264286 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:14.264292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:14.265653 (+ 1361us) server_negotiation.cc:407] Connection header received
0621 07:05:14.268308 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:14.268317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:14.268430 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:14.268578 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:14.270178 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.271175 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.272957 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.273487 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.273618 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:14.275290 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:14.275295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:14.275298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:14.275319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:14.275360 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:14.275373 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:14.275552 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:14.275751 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 07:05:14.275845 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260621 07:05:14.277384 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:14.264265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:39697 (local address 127.1.188.195:60345)
0621 07:05:14.265421 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:14.265483 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:14.267910 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:14.268828 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:14.268838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:14.268998 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:14.269976 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:14.270002 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.271313 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.271317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:14.272817 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:14.272829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.273737 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.273742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:14.273981 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:14.274886 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:14.274892 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:14.274960 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:14.275088 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:14.275543 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:14.275551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:14.275557 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:05:14.275726 (+ 169us) client_negotiation.cc:240] Negotiation successful
0621 07:05:14.276112 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":345,"threads_started":1}
I20260621 07:05:14.278890 11575 heartbeater.cc:347] Connected to a master server at 127.1.188.254:39697
I20260621 07:05:14.279289 11575 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:14.280295 11575 heartbeater.cc:511] Master 127.1.188.254:39697 requested a full tablet report, sending...
I20260621 07:05:14.282202 11126 ts_manager.cc:194] Registered new tserver with Master: f46f9a26f892433bae8c97d09b46a750 (127.1.188.195:35589)
I20260621 07:05:14.283300 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:05:14.283398 11126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.195:60345
W20260621 07:05:14.290608 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:14.288839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.1.188.254:39697)
0621 07:05:14.289047 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:14.289053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:14.289509 (+ 456us) server_negotiation.cc:407] Connection header received
0621 07:05:14.289669 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:14.289673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:14.289746 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:14.289843 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:14.290241 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40462: BlockingRecv error: recv got EOF from 127.0.0.1:40462 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 07:05:14.291509 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11186
I20260621 07:05:14.300956 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:14.292610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.1.188.193:40573)
0621 07:05:14.293382 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:14.293400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:14.293434 (+ 34us) server_negotiation.cc:407] Connection header received
0621 07:05:14.293504 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:14.293511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:14.293784 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:14.293933 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:14.294395 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.295643 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.297146 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.297917 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.298020 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:14.299201 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:14.299212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:14.299227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:14.300047 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:14.300103 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:14.300125 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:14.300267 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:14.300444 (+ 177us) server_negotiation.cc:299] Negotiation successful
0621 07:05:14.300594 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":294,"threads_started":1}
I20260621 07:05:14.302462 11308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:14.388695 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11186
I20260621 07:05:14.406091 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11317
I20260621 07:05:14.414487 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:14.407036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.1.188.194:38549)
0621 07:05:14.407766 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:14.407774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:14.407807 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:05:14.407884 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:14.407891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:14.408146 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:14.408288 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:14.408668 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.409724 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.411162 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.411792 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.411893 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:14.412936 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:14.412944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:14.412960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:14.413639 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:14.413688 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:14.413715 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:14.413812 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:14.414035 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 07:05:14.414183 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":356,"threads_started":1}
I20260621 07:05:14.415910 11439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:14.505183 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11317
I20260621 07:05:14.521252 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11448
I20260621 07:05:14.529456 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:14.522182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.1.188.195:35589)
0621 07:05:14.522835 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:14.522848 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:14.522881 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:05:14.522950 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:14.522956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:14.523182 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:14.523354 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:14.523843 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.524911 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.526174 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:14.526728 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:14.526873 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:14.528005 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:14.528013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:14.528026 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:14.528612 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:14.528656 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:14.528675 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:14.528802 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:14.528972 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 07:05:14.529099 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":327,"threads_started":1}
I20260621 07:05:14.530709 11570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:14.617072 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11448
I20260621 07:05:14.632893 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11096
I20260621 07:05:14.634140 11157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:14.738605 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11096
2026-06-21T07:05:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2818 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T07:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:05:14Z Disabled control of system clock
I20260621 07:05:14.811568 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37065
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:05:15.156296 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:15.156586 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:15.156773 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:15.164078 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:05:15.164182 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:15.164250 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:15.164358 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:05:15.164436 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:05:15.173576 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:15.174883 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:15.176744 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:15.188370 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:15.188445 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:15.188421 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:15.189415 11595 server_base.cc:1034] running on GCE node
I20260621 07:05:15.190148 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:15.191591 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:15.192795 11595 hybrid_clock.cc:648] HybridClock initialized: now 1782025515192746 us; error 39 us; skew 500 ppm
I20260621 07:05:15.193149 11595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:15.195350 11595 webserver.cc:460] Webserver started at http://127.1.188.254:38411/ using document root <none> and password file <none>
I20260621 07:05:15.196081 11595 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:15.196203 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:15.196535 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:15.200898 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "58e26853c6904d0ea999cce10c881b40"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "26d5035b61a3666d437069f536db78d6"
server_key_iv: "643c52b94ad9afc74779fe25f3e2289d"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.201728 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "58e26853c6904d0ea999cce10c881b40"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "26d5035b61a3666d437069f536db78d6"
server_key_iv: "643c52b94ad9afc74779fe25f3e2289d"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.206810 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 07:05:15.210306 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:15.211812 11595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:05:15.211988 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "58e26853c6904d0ea999cce10c881b40"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "26d5035b61a3666d437069f536db78d6"
server_key_iv: "643c52b94ad9afc74779fe25f3e2289d"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.212181 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:15.217823 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:15.218683 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:15.219070 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:15.237989 11595 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:37065
I20260621 07:05:15.238019 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:37065 every 8 connection(s)
I20260621 07:05:15.239342 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:05:15.242539 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11595
I20260621 07:05:15.242789 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:05:15.243516 1779 external_mini_cluster.cc:1428] Setting key 0cff29714b894c47695a43df1cf152fc
I20260621 07:05:15.243579 11661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:15.252081 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:15.245654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.1.188.254:37065)
0621 07:05:15.246711 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:15.246721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:15.246760 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:05:15.248553 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:15.248577 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:15.248960 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:15.249405 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:15.250445 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:15.250451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:15.250462 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:15.250476 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:15.250528 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:15.250542 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:15.250785 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:15.250933 (+ 148us) server_negotiation.cc:299] Negotiation successful
0621 07:05:15.251171 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":410,"threads_started":1}
I20260621 07:05:15.257248 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40: Bootstrap starting.
I20260621 07:05:15.260587 11661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:15.261639 11661 log.cc:824] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:15.264607 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40: No bootstrap required, opened a new log
I20260621 07:05:15.272836 11661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } }
I20260621 07:05:15.273373 11661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:15.273576 11661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e26853c6904d0ea999cce10c881b40, State: Initialized, Role: FOLLOWER
I20260621 07:05:15.274302 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } }
I20260621 07:05:15.274521 11661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:05:15.274639 11661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:05:15.274775 11661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:15.277180 11661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } }
I20260621 07:05:15.277680 11661 leader_election.cc:302] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e26853c6904d0ea999cce10c881b40; no voters:
I20260621 07:05:15.278316 11661 leader_election.cc:288] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:05:15.278497 11666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:05:15.279222 11666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [term 1 LEADER]: Becoming Leader. State: Replica: 58e26853c6904d0ea999cce10c881b40, State: Running, Role: LEADER
I20260621 07:05:15.279780 11666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } }
I20260621 07:05:15.280673 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:05:15.282518 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e26853c6904d0ea999cce10c881b40. Latest consensus state: current_term: 1 leader_uuid: "58e26853c6904d0ea999cce10c881b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } } }
I20260621 07:05:15.282907 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:15.283160 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e26853c6904d0ea999cce10c881b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e26853c6904d0ea999cce10c881b40" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37065 } } }
I20260621 07:05:15.283666 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:15.285184 11672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:05:15.290151 11672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:05:15.299198 11672 catalog_manager.cc:1347] Generated new cluster ID: 77abf971a8124af282b07bba5febbbcd
I20260621 07:05:15.299307 11672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:05:15.330283 11672 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:05:15.330737 11672 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:05:15.346037 11672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e26853c6904d0ea999cce10c881b40: Generated new TSK 0
I20260621 07:05:15.347007 11672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:05:15.358986 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--builtin_ntp_servers=127.1.188.212:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:05:15.705338 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:15.705606 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:15.705778 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:15.713974 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:15.714067 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:15.714282 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:05:15.723850 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:15.725126 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:15.726878 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:15.740087 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:15.740069 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:15.740691 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:15.746922 11685 server_base.cc:1034] running on GCE node
I20260621 07:05:15.747771 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:15.748797 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:15.750046 11685 hybrid_clock.cc:648] HybridClock initialized: now 1782025515749999 us; error 47 us; skew 500 ppm
I20260621 07:05:15.750427 11685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:15.752826 11685 webserver.cc:460] Webserver started at http://127.1.188.193:39669/ using document root <none> and password file <none>
I20260621 07:05:15.753537 11685 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:15.753690 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:15.754019 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:15.756862 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "d3abf6dba1504fc0865da14b14fe3323"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "da8ab5b2b4a3952eda6a9797778667ad"
server_key_iv: "590ca3bce6139a22d9ef2f671b8f4b1e"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.757653 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "d3abf6dba1504fc0865da14b14fe3323"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "da8ab5b2b4a3952eda6a9797778667ad"
server_key_iv: "590ca3bce6139a22d9ef2f671b8f4b1e"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.762681 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 07:05:15.766215 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:15.767961 11685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:05:15.768169 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "d3abf6dba1504fc0865da14b14fe3323"
format_stamp: "Formatted at 2026-06-21 07:05:15 on dist-test-slave-hlq1"
server_key: "da8ab5b2b4a3952eda6a9797778667ad"
server_key_iv: "590ca3bce6139a22d9ef2f671b8f4b1e"
server_key_version: "encryptionkey@0"
I20260621 07:05:15.768391 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:15.774101 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:15.774889 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:15.775161 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:15.776162 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:15.777608 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:15.777740 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:15.777899 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:15.777990 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:15.811901 11685 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:42475
I20260621 07:05:15.811988 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:42475 every 8 connection(s)
I20260621 07:05:15.813233 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:05:15.815188 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11685
I20260621 07:05:15.815423 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:05:15.816220 1779 external_mini_cluster.cc:1428] Setting key f0a09f989e89bf04f040bdbd5dac4d87
I20260621 07:05:15.819502 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--builtin_ntp_servers=127.1.188.212:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 07:05:15.822958 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:15.816223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:58667 (local address 127.1.188.254:37065)
0621 07:05:15.816696 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:15.816702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:15.818135 (+ 1433us) server_negotiation.cc:407] Connection header received
0621 07:05:15.820679 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:15.820683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:15.820733 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:15.820823 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:15.822313 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:15.822318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:15.822321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:15.822336 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:15.822375 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:15.822387 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:15.822483 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:15.822707 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 07:05:15.822754 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260621 07:05:15.824328 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:15.816845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37065 (local address 127.1.188.193:58667)
0621 07:05:15.817967 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:15.818027 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:15.820382 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:15.821094 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:15.821109 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:15.821238 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:15.821977 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:15.821983 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:15.822034 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:15.822141 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:15.822485 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:15.822497 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:15.822508 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:05:15.822784 (+ 276us) client_negotiation.cc:240] Negotiation successful
0621 07:05:15.823035 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260621 07:05:15.825877 11812 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37065
I20260621 07:05:15.826171 11812 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:15.826756 11812 heartbeater.cc:511] Master 127.1.188.254:37065 requested a full tablet report, sending...
I20260621 07:05:15.828914 11626 ts_manager.cc:194] Registered new tserver with Master: d3abf6dba1504fc0865da14b14fe3323 (127.1.188.193:42475)
W20260621 07:05:16.142640 11816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:16.142983 11816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:16.143150 11816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:16.150593 11816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:16.150741 11816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:16.150929 11816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:05:16.160327 11816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:16.161681 11816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:16.163591 11816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:16.176786 11821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:16.176824 11824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:16.176852 11822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:16.177700 11816 server_base.cc:1034] running on GCE node
I20260621 07:05:16.178298 11816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:16.179214 11816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:16.180435 11816 hybrid_clock.cc:648] HybridClock initialized: now 1782025516180366 us; error 59 us; skew 500 ppm
I20260621 07:05:16.180802 11816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:16.183334 11816 webserver.cc:460] Webserver started at http://127.1.188.194:41973/ using document root <none> and password file <none>
I20260621 07:05:16.184078 11816 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:16.184167 11816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:16.184541 11816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:16.187361 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "1002e0f45b8a4e259e14a3b061301103"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "cc7342c0b433e5d8a7a3f2ea5b7aaf46"
server_key_iv: "579bc74efd892644e974240b0e846aaf"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.188256 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "1002e0f45b8a4e259e14a3b061301103"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "cc7342c0b433e5d8a7a3f2ea5b7aaf46"
server_key_iv: "579bc74efd892644e974240b0e846aaf"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.193365 11816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:05:16.196878 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.198217 11816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:05:16.198390 11816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "1002e0f45b8a4e259e14a3b061301103"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "cc7342c0b433e5d8a7a3f2ea5b7aaf46"
server_key_iv: "579bc74efd892644e974240b0e846aaf"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.198613 11816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:16.204427 11816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:16.205211 11816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:16.205482 11816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:16.206387 11816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:16.207888 11816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:16.208017 11816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.208158 11816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:16.208245 11816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.242141 11816 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:46033
I20260621 07:05:16.242164 11942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:46033 every 8 connection(s)
I20260621 07:05:16.243520 11816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:05:16.251807 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.246227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:36631 (local address 127.1.188.254:37065)
0621 07:05:16.246398 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.246403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.247878 (+ 1475us) server_negotiation.cc:407] Connection header received
0621 07:05:16.249550 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.249555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.249645 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.249751 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.251033 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:16.251038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:16.251043 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:16.251080 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:16.251109 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:16.251120 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:16.251242 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:16.251447 (+ 205us) server_negotiation.cc:299] Negotiation successful
0621 07:05:16.251515 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 07:05:16.252691 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.246731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37065 (local address 127.1.188.194:36631)
0621 07:05:16.247725 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:16.247765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:16.249277 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:16.249970 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:16.249978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:16.250066 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:16.250756 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:16.250758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:16.250793 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:16.250872 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:16.251215 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:16.251225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:16.251233 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:05:16.251533 (+ 300us) client_negotiation.cc:240] Negotiation successful
0621 07:05:16.251791 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":303,"threads_started":1}
I20260621 07:05:16.253909 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11816
I20260621 07:05:16.254166 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:05:16.254184 11943 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37065
I20260621 07:05:16.254477 11943 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:16.254748 1779 external_mini_cluster.cc:1428] Setting key e65968ea9e19cff28d89d8c07150856c
I20260621 07:05:16.255179 11943 heartbeater.cc:511] Master 127.1.188.254:37065 requested a full tablet report, sending...
I20260621 07:05:16.256716 11625 ts_manager.cc:194] Registered new tserver with Master: 1002e0f45b8a4e259e14a3b061301103 (127.1.188.194:46033)
I20260621 07:05:16.257623 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--builtin_ntp_servers=127.1.188.212:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 07:05:16.583758 11946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:16.584048 11946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:16.584146 11946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:16.591809 11946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:16.591909 11946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:16.592017 11946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:05:16.601400 11946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:37065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:16.603009 11946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:16.604827 11946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:16.618142 11951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:16.618184 11954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:16.618166 11952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:16.620970 11946 server_base.cc:1034] running on GCE node
I20260621 07:05:16.621711 11946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:16.622797 11946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:16.624045 11946 hybrid_clock.cc:648] HybridClock initialized: now 1782025516623991 us; error 43 us; skew 500 ppm
I20260621 07:05:16.624439 11946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:05:16.626801 11946 webserver.cc:460] Webserver started at http://127.1.188.195:39279/ using document root <none> and password file <none>
I20260621 07:05:16.627575 11946 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:16.627704 11946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:16.628032 11946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:16.630873 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "a40f3b6e49b344a4811bc26312f071bb"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "a8626f87fd3db981a5e2f2c1516e5b02"
server_key_iv: "ea7b11334f66f142c21a4fe4fa997261"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.631728 11946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "a40f3b6e49b344a4811bc26312f071bb"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "a8626f87fd3db981a5e2f2c1516e5b02"
server_key_iv: "ea7b11334f66f142c21a4fe4fa997261"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.636758 11946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:05:16.640323 11960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.641645 11946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:05:16.641824 11946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "a40f3b6e49b344a4811bc26312f071bb"
format_stamp: "Formatted at 2026-06-21 07:05:16 on dist-test-slave-hlq1"
server_key: "a8626f87fd3db981a5e2f2c1516e5b02"
server_key_iv: "ea7b11334f66f142c21a4fe4fa997261"
server_key_version: "encryptionkey@0"
I20260621 07:05:16.642048 11946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:16.648309 11946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:16.649405 11946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:16.649691 11946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:16.651119 11946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:16.653280 11946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:16.653399 11946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.653517 11946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:16.653601 11946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:16.692530 11946 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:41187
I20260621 07:05:16.692670 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:41187 every 8 connection(s)
I20260621 07:05:16.693945 11946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:05:16.699834 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 11946
I20260621 07:05:16.700014 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:05:16.700587 1779 external_mini_cluster.cc:1428] Setting key 824845add71793ab8fc8d8eb7b447128
I20260621 07:05:16.704444 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.696944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:40471 (local address 127.1.188.254:37065)
0621 07:05:16.697163 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.697172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.698992 (+ 1820us) server_negotiation.cc:407] Connection header received
0621 07:05:16.701794 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.701799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.701866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.701959 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.703682 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:16.703688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:16.703693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:16.703713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:16.703754 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:16.703766 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:16.703884 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:16.704093 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 07:05:16.704167 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:05:16.705360 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.697630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37065 (local address 127.1.188.195:40471)
0621 07:05:16.698810 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:16.698869 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:16.701479 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:16.702298 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:16.702313 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:16.702429 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:16.703326 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:05:16.703331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:16.703387 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:05:16.703523 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:16.703868 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:16.703877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:16.703884 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:05:16.704171 (+ 287us) client_negotiation.cc:240] Negotiation successful
0621 07:05:16.704411 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":337,"threads_started":1}
I20260621 07:05:16.706629 12073 heartbeater.cc:347] Connected to a master server at 127.1.188.254:37065
I20260621 07:05:16.706895 12073 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:16.707521 12073 heartbeater.cc:511] Master 127.1.188.254:37065 requested a full tablet report, sending...
I20260621 07:05:16.708981 11625 ts_manager.cc:194] Registered new tserver with Master: a40f3b6e49b344a4811bc26312f071bb (127.1.188.195:41187)
I20260621 07:05:16.716707 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:05:16.723372 11814 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:05:16.721659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.1.188.254:37065)
0621 07:05:16.721842 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.721847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.722385 (+ 538us) server_negotiation.cc:407] Connection header received
0621 07:05:16.722513 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.722517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.722571 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.722662 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.723031 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43470: BlockingRecv error: recv got EOF from 127.0.0.1:43470 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:05:16.724217 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11685
I20260621 07:05:16.727946 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.725158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.1.188.193:42475)
0621 07:05:16.725743 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.725751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.725777 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:05:16.725836 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.725842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.726050 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.726266 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.727226 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:16.727233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:16.727244 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:16.727256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:16.727294 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:16.727308 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:16.727476 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:16.727582 (+ 106us) server_negotiation.cc:299] Negotiation successful
0621 07:05:16.727690 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
I20260621 07:05:16.729144 11807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:16.809993 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11685
I20260621 07:05:16.825994 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11816
I20260621 07:05:16.829823 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.827052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.1.188.194:46033)
0621 07:05:16.827679 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.827687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.827715 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:05:16.827777 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.827783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.828008 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.828132 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.829087 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:16.829093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:16.829104 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:16.829122 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:16.829194 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:16.829209 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:16.829312 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:16.829430 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 07:05:16.829539 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":310,"threads_started":1}
I20260621 07:05:16.831041 11938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:16.912664 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11816
I20260621 07:05:16.929322 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11946
I20260621 07:05:16.933101 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:16.930294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.1.188.195:41187)
0621 07:05:16.930896 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:16.930902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:16.930933 (+ 31us) server_negotiation.cc:407] Connection header received
0621 07:05:16.930995 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:16.931002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:16.931247 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:16.931377 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:16.932391 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:16.932398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:16.932409 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:05:16.932423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:16.932467 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:05:16.932484 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:16.932628 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:16.932675 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:05:16.932790 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":307,"threads_started":1}
I20260621 07:05:16.934394 12068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:17.015632 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11946
I20260621 07:05:17.031214 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 11595
I20260621 07:05:17.032511 11656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:17.136483 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 11595
2026-06-21T07:05:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2396 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:05:17 dist-test-slave-hlq1 krb5kdc[12091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:05:17 dist-test-slave-hlq1 krb5kdc[12091](info): set up 2 sockets
Jun 21 07:05:17 dist-test-slave-hlq1 krb5kdc[12091](info): commencing operation
krb5kdc: starting...
W20260621 07:05:18.245144 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:05:18 dist-test-slave-hlq1 krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:05:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:05:18.514796 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35805
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:05:18.911866 12104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:18.912232 12104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:18.912408 12104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:18.920369 12104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:05:18.920512 12104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:18.920639 12104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:18.920755 12104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:05:18.920835 12104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:05:18.931916 12104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:35805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:18.933779 12104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:18.935621 12104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:18.950106 12109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:18.950079 12110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:18.949913 12112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:18.950798 12104 server_base.cc:1034] running on GCE node
I20260621 07:05:18.951876 12104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:18.953635 12104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:18.954870 12104 hybrid_clock.cc:648] HybridClock initialized: now 1782025518954811 us; error 40 us; skew 500 ppm
I20260621 07:05:18.955291 12104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:05:18 dist-test-slave-hlq1 krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:05:18.964295 12104 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:05:18.966367 12104 webserver.cc:460] Webserver started at http://127.1.188.254:38915/ using document root <none> and password file <none>
I20260621 07:05:18.967221 12104 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:18.967373 12104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:18.967757 12104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:18.971204 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "0690cb071df046e78bfaac42dfce8c55"
format_stamp: "Formatted at 2026-06-21 07:05:18 on dist-test-slave-hlq1"
server_key: "6a9872ae4ccfe211a76971546c04ef4f"
server_key_iv: "464e18d1c2aac1d6e5a926e491a2f6c1"
server_key_version: "encryptionkey@0"
I20260621 07:05:18.972121 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "0690cb071df046e78bfaac42dfce8c55"
format_stamp: "Formatted at 2026-06-21 07:05:18 on dist-test-slave-hlq1"
server_key: "6a9872ae4ccfe211a76971546c04ef4f"
server_key_iv: "464e18d1c2aac1d6e5a926e491a2f6c1"
server_key_version: "encryptionkey@0"
I20260621 07:05:18.978212 12104 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260621 07:05:18.982113 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:18.983736 12104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:05:18.983929 12104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "0690cb071df046e78bfaac42dfce8c55"
format_stamp: "Formatted at 2026-06-21 07:05:18 on dist-test-slave-hlq1"
server_key: "6a9872ae4ccfe211a76971546c04ef4f"
server_key_iv: "464e18d1c2aac1d6e5a926e491a2f6c1"
server_key_version: "encryptionkey@0"
I20260621 07:05:18.984155 12104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:19.000828 12104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:19.005503 12104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:19.005923 12104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:19.028159 12104 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:35805
I20260621 07:05:19.028223 12170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:35805 every 8 connection(s)
I20260621 07:05:19.029920 12104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:05:19.034860 12171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:19.035213 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 12104
I20260621 07:05:19.035509 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:05:19.036311 1779 external_mini_cluster.cc:1428] Setting key 40b2588466e5c83b8d435b7e462ec565
I20260621 07:05:19.051167 12171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: Bootstrap starting.
Jun 21 07:05:19 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:05:19.055312 12171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:19.056939 12171 log.cc:824] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:19.061115 12171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: No bootstrap required, opened a new log
I20260621 07:05:19.064450 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:19.038729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.1.188.254:35805)
0621 07:05:19.039651 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:19.039671 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:19.039731 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:05:19.042609 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:19.042644 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:19.043259 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:19.043810 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:19.045321 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.047482 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.048925 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.050012 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.050210 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:19.055134 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:19.055149 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:19.055180 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:19.055236 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:19.059469 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:19.060191 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:19.060200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:19.060209 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:19.060493 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:19.060860 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:19.060864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:19.060867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:19.061674 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:19.062032 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:19.062348 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 07:05:19.062836 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":345,"threads_started":1}
I20260621 07:05:19.071175 12171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } }
I20260621 07:05:19.071697 12171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:19.071780 12171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0690cb071df046e78bfaac42dfce8c55, State: Initialized, Role: FOLLOWER
I20260621 07:05:19.072343 12171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } }
I20260621 07:05:19.072556 12171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:05:19.072655 12171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:05:19.072788 12171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:19.075151 12171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } }
I20260621 07:05:19.075824 12171 leader_election.cc:302] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0690cb071df046e78bfaac42dfce8c55; no voters:
I20260621 07:05:19.076596 12171 leader_election.cc:288] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:05:19.076797 12176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:05:19.077629 12176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [term 1 LEADER]: Becoming Leader. State: Replica: 0690cb071df046e78bfaac42dfce8c55, State: Running, Role: LEADER
I20260621 07:05:19.078281 12176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } }
I20260621 07:05:19.078964 12171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:05:19.082093 12178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0690cb071df046e78bfaac42dfce8c55. Latest consensus state: current_term: 1 leader_uuid: "0690cb071df046e78bfaac42dfce8c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } } }
I20260621 07:05:19.082055 12177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0690cb071df046e78bfaac42dfce8c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0690cb071df046e78bfaac42dfce8c55" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35805 } } }
I20260621 07:05:19.082518 12178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:19.082612 12177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55 [sys.catalog]: This master's current role is: LEADER
I20260621 07:05:19.084275 12184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:05:19.090518 12184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:05:19.100731 12184 catalog_manager.cc:1347] Generated new cluster ID: c7bd2843d5ac475999cd788e37956eb1
I20260621 07:05:19.100873 12184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:05:19.130342 12184 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:05:19.132645 12184 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:05:19.143294 12184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: Generated new TSK 0
I20260621 07:05:19.144665 12184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:05:19.209596 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:05:19.589811 12197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:19.590118 12197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:19.590287 12197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:19.597733 12197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:19.597862 12197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:19.598042 12197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:05:19.608397 12197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:19.609992 12197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:19.611539 12197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:19.626082 12203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:19.626276 12205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:19.626876 12202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:19.627094 12197 server_base.cc:1034] running on GCE node
I20260621 07:05:19.627919 12197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:19.629081 12197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:19.630295 12197 hybrid_clock.cc:648] HybridClock initialized: now 1782025519630240 us; error 49 us; skew 500 ppm
I20260621 07:05:19.630669 12197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:05:19 dist-test-slave-hlq1 krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025519, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:05:19.634092 12197 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:05:19.635852 12197 webserver.cc:460] Webserver started at http://127.1.188.193:36565/ using document root <none> and password file <none>
I20260621 07:05:19.636633 12197 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:19.636790 12197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:19.637121 12197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:19.640307 12197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "6bf4647e38fb4595abfae2ca8cc23c86"
format_stamp: "Formatted at 2026-06-21 07:05:19 on dist-test-slave-hlq1"
server_key: "f4b2acca507b30dc1aeb019a09ac45ef"
server_key_iv: "50803a8a035f313fa663a4e4a9260126"
server_key_version: "encryptionkey@0"
I20260621 07:05:19.641098 12197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "6bf4647e38fb4595abfae2ca8cc23c86"
format_stamp: "Formatted at 2026-06-21 07:05:19 on dist-test-slave-hlq1"
server_key: "f4b2acca507b30dc1aeb019a09ac45ef"
server_key_iv: "50803a8a035f313fa663a4e4a9260126"
server_key_version: "encryptionkey@0"
I20260621 07:05:19.646646 12197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:05:19.650241 12212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:19.651770 12197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:05:19.651947 12197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "6bf4647e38fb4595abfae2ca8cc23c86"
format_stamp: "Formatted at 2026-06-21 07:05:19 on dist-test-slave-hlq1"
server_key: "f4b2acca507b30dc1aeb019a09ac45ef"
server_key_iv: "50803a8a035f313fa663a4e4a9260126"
server_key_version: "encryptionkey@0"
I20260621 07:05:19.652148 12197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:19.692904 12197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:19.697664 12197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:19.697973 12197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:19.699016 12197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:19.700484 12197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:19.700589 12197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:19.700740 12197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:19.700814 12197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:19.734791 12197 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33669
I20260621 07:05:19.734833 12324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33669 every 8 connection(s)
I20260621 07:05:19.736198 12197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:05:19.742026 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 12197
I20260621 07:05:19.742273 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:05:19.743151 1779 external_mini_cluster.cc:1428] Setting key de9886e07a511af630c12bb023866fc5
Jun 21 07:05:19 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025519, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:05:19.758916 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:19.738922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:33727 (local address 127.1.188.254:35805)
0621 07:05:19.739523 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:19.739528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:19.740355 (+ 827us) server_negotiation.cc:407] Connection header received
0621 07:05:19.742202 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:19.742207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:19.742306 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:19.742443 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:19.744493 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.745566 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.748175 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.748628 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.748737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:19.752985 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:19.752990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:19.752994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:19.753027 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:19.755747 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:19.756502 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:19.756507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:19.756513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:19.756636 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:19.757145 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:19.757154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:19.757157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:19.757415 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:19.757587 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:19.758296 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 07:05:19.758521 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":317,"threads_started":1}
I20260621 07:05:19.759989 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:19.739488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35805 (local address 127.1.188.193:33727)
0621 07:05:19.740194 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:19.740233 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:19.741910 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:19.742728 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:19.742742 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:19.743539 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:19.744327 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:19.744348 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.745745 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.745750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:19.747949 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:19.747968 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:19.748870 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:19.748879 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:19.749194 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:19.749973 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:19.750006 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:19.752746 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:19.755898 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:19.755907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:19.755925 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:19.756370 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:19.756754 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:19.756760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:19.756763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:19.757026 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:19.757554 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:19.757562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:19.757914 (+ 352us) client_negotiation.cc:769] Sending connection context
0621 07:05:19.758193 (+ 279us) client_negotiation.cc:240] Negotiation successful
0621 07:05:19.758730 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260621 07:05:19.761644 12325 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35805
I20260621 07:05:19.761983 12325 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:19.762780 12325 heartbeater.cc:511] Master 127.1.188.254:35805 requested a full tablet report, sending...
I20260621 07:05:19.765051 12136 ts_manager.cc:194] Registered new tserver with Master: 6bf4647e38fb4595abfae2ca8cc23c86 (127.1.188.193:33669)
I20260621 07:05:19.766606 12136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:33727
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:05:19.794552 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:05:20.150698 12331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:20.151006 12331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:20.151208 12331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:20.159634 12331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:20.159765 12331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:20.159957 12331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:05:20.170544 12331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:20.171898 12331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:20.173173 12331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:20.186329 12336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:20.186306 12337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:20.186306 12339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:20.187095 12331 server_base.cc:1034] running on GCE node
I20260621 07:05:20.187750 12331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:20.188776 12331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:20.190001 12331 hybrid_clock.cc:648] HybridClock initialized: now 1782025520189960 us; error 35 us; skew 500 ppm
I20260621 07:05:20.190317 12331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:05:20 dist-test-slave-hlq1 krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:05:20.193599 12331 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:05:20.195026 12331 webserver.cc:460] Webserver started at http://127.1.188.194:40731/ using document root <none> and password file <none>
I20260621 07:05:20.195832 12331 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:20.195964 12331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:20.196259 12331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:20.199187 12331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "d2bc8073d3834855aed4b18839cff667"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "07ded54b038a24f9bcb61dd6d81ba685"
server_key_iv: "94acb6e9730c81e4a5a2eb746142d82e"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.200098 12331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "d2bc8073d3834855aed4b18839cff667"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "07ded54b038a24f9bcb61dd6d81ba685"
server_key_iv: "94acb6e9730c81e4a5a2eb746142d82e"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.205070 12331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:05:20.208492 12346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.209793 12331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:05:20.209957 12331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "d2bc8073d3834855aed4b18839cff667"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "07ded54b038a24f9bcb61dd6d81ba685"
server_key_iv: "94acb6e9730c81e4a5a2eb746142d82e"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.210155 12331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:20.230645 12331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:20.234644 12331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:20.234925 12331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:20.235916 12331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:20.237219 12331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:20.237326 12331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.237473 12331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:20.237545 12331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.271633 12331 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:35937
I20260621 07:05:20.271677 12458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:35937 every 8 connection(s)
I20260621 07:05:20.272866 12331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:05:20.277168 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 12331
I20260621 07:05:20.277365 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:05:20.277789 1779 external_mini_cluster.cc:1428] Setting key 2df4ff6129a00ed3969c37fcf2318caf
Jun 21 07:05:20 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:05:20.293365 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.275380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:60477 (local address 127.1.188.254:35805)
0621 07:05:20.275833 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.275837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.276807 (+ 970us) server_negotiation.cc:407] Connection header received
0621 07:05:20.278956 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.278962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.279053 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.279237 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:20.280877 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.281594 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.283068 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.283572 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.283709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.287457 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:20.287464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:20.287469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:20.287511 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:20.290161 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.291052 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.291065 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.291068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.291212 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.291704 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.291707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.291709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.291944 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:20.292048 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.292924 (+ 876us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.293074 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":213,"threads_started":1}
I20260621 07:05:20.294822 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.275985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35805 (local address 127.1.188.194:60477)
0621 07:05:20.276658 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:20.276699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:20.278674 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:20.279487 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:20.279499 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:20.280102 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:20.280717 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.280733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.281743 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.281749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.282926 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.282937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.283795 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.283805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.284070 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.284804 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:20.284830 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:20.287126 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:20.290295 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:20.290306 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:20.290324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:20.290936 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:20.291317 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:20.291324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:20.291331 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:20.291592 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:20.292084 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:20.292094 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:20.292512 (+ 418us) client_negotiation.cc:769] Sending connection context
0621 07:05:20.292827 (+ 315us) client_negotiation.cc:240] Negotiation successful
0621 07:05:20.293397 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260621 07:05:20.296561 12459 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35805
I20260621 07:05:20.296876 12459 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:20.297796 12459 heartbeater.cc:511] Master 127.1.188.254:35805 requested a full tablet report, sending...
I20260621 07:05:20.299566 12136 ts_manager.cc:194] Registered new tserver with Master: d2bc8073d3834855aed4b18839cff667 (127.1.188.194:35937)
I20260621 07:05:20.300387 12136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:60477
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:05:20.330885 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:05:20.673287 12465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:05:20.673671 12465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:05:20.673848 12465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:05:20.681785 12465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:05:20.681960 12465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:05:20.682149 12465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:05:20.692945 12465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:05:20.694566 12465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:05:20.696187 12465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:05:20.710625 12473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:20.710721 12470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:05:20.710728 12471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:05:20.712005 12465 server_base.cc:1034] running on GCE node
I20260621 07:05:20.712654 12465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:05:20.713728 12465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:05:20.714967 12465 hybrid_clock.cc:648] HybridClock initialized: now 1782025520714908 us; error 41 us; skew 500 ppm
I20260621 07:05:20.715312 12465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:05:20 dist-test-slave-hlq1 krb5kdc[12091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:05:20.718958 12465 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:05:20.720614 12465 webserver.cc:460] Webserver started at http://127.1.188.195:33143/ using document root <none> and password file <none>
I20260621 07:05:20.721392 12465 fs_manager.cc:359] Metadata directory not provided
I20260621 07:05:20.721508 12465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:05:20.721799 12465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:05:20.725309 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "07a2f2077933422c9d74f643142dcee5"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "3f8c8618c1d9c9c145d974ea2e93d07c"
server_key_iv: "f4a2f90562c2fe611c4e8e693aadc39c"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.726152 12465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "07a2f2077933422c9d74f643142dcee5"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "3f8c8618c1d9c9c145d974ea2e93d07c"
server_key_iv: "f4a2f90562c2fe611c4e8e693aadc39c"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.732331 12465 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260621 07:05:20.736950 12480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.738701 12465 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:05:20.738890 12465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "07a2f2077933422c9d74f643142dcee5"
format_stamp: "Formatted at 2026-06-21 07:05:20 on dist-test-slave-hlq1"
server_key: "3f8c8618c1d9c9c145d974ea2e93d07c"
server_key_iv: "f4a2f90562c2fe611c4e8e693aadc39c"
server_key_version: "encryptionkey@0"
I20260621 07:05:20.739130 12465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:05:20.770085 12325 heartbeater.cc:503] Master 127.1.188.254:35805 was elected leader, sending a full tablet report...
I20260621 07:05:20.775974 12465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:05:20.780733 12465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:05:20.781096 12465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:05:20.782315 12465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:05:20.783917 12465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:05:20.784005 12465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.784112 12465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:05:20.784158 12465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:05:20.820537 12465 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:37413
I20260621 07:05:20.820578 12592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:37413 every 8 connection(s)
I20260621 07:05:20.822077 12465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:05:20.824729 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 12465
I20260621 07:05:20.824963 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:05:20.825609 1779 external_mini_cluster.cc:1428] Setting key 15a6ac32ebf3e3eb6ff35ec004b9fa56
Jun 21 07:05:20 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025520, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:05:20.846321 12596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.826486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:54785 (local address 127.1.188.254:35805)
0621 07:05:20.827051 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.827058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.829942 (+ 2884us) server_negotiation.cc:407] Connection header received
0621 07:05:20.830046 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.830052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.830225 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.830379 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:20.832313 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.833198 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.834937 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.835362 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.835549 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.839635 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:20.839640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:20.839643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:20.839680 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:20.843029 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.844077 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.844082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.844085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.844224 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.844643 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.844647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.844649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.844941 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:20.845089 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.845825 (+ 736us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.846013 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
I20260621 07:05:20.847529 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.826039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35805 (local address 127.1.188.195:54785)
0621 07:05:20.827059 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:20.827112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:20.829583 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:20.830718 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:20.830729 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:20.831383 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:05:20.832129 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.832149 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.833375 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.833380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.834787 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.834800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.835677 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.835683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.835917 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.836733 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:05:20.836766 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:05:20.839322 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:05:20.843198 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:20.843208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:20.843231 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:20.843952 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:20.844318 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:05:20.844321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:05:20.844324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:05:20.844544 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:05:20.845052 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:05:20.845058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:05:20.845465 (+ 407us) client_negotiation.cc:769] Sending connection context
0621 07:05:20.845721 (+ 256us) client_negotiation.cc:240] Negotiation successful
0621 07:05:20.846180 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":435,"threads_started":1}
I20260621 07:05:20.849011 12593 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35805
I20260621 07:05:20.849334 12593 heartbeater.cc:464] Registering TS with master...
I20260621 07:05:20.850152 12593 heartbeater.cc:511] Master 127.1.188.254:35805 requested a full tablet report, sending...
I20260621 07:05:20.851742 12136 ts_manager.cc:194] Registered new tserver with Master: 07a2f2077933422c9d74f643142dcee5 (127.1.188.195:37413)
I20260621 07:05:20.852674 12136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:54785
I20260621 07:05:20.855810 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:05:20.874429 12596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.861009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.1.188.254:35805)
0621 07:05:20.861258 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.861269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.861661 (+ 392us) server_negotiation.cc:407] Connection header received
0621 07:05:20.861801 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.861805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.861887 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.861969 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:20.863544 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.864320 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.865596 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.865993 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.866085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.868615 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:20.868621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:20.868624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:20.868659 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:20.871524 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.872287 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.872291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.872294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.872445 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:20.872863 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:20.872867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:20.872871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:20.873254 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:20.873453 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.873828 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.874029 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:05:20.879778 12136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:05:20.887944 12136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:05:20.928548 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.902514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55028 (local address 127.1.188.195:37413)
0621 07:05:20.919325 (+ 16811us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.919340 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.919430 (+ 90us) server_negotiation.cc:407] Connection header received
0621 07:05:20.919575 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.919589 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.919997 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.920890 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:05:20.921230 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.922603 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.925965 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.927154 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.927320 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.927661 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.927898 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.928082 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16537,"thread_start_us":16780,"threads_started":1}
I20260621 07:05:20.928730 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.902515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:35937 (local address 127.0.0.1:46604)
0621 07:05:20.917120 (+ 14605us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:20.917144 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:20.917304 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:20.921153 (+ 3849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:20.921161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:20.921189 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:05:20.921664 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.921678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.923817 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.923824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.925887 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.925898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.927802 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.927812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.927994 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.928029 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:05:20.928090 (+ 61us) client_negotiation.cc:240] Negotiation successful
0621 07:05:20.928231 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14435,"mutex_wait_us":81,"thread_start_us":11363,"threads_started":1}
I20260621 07:05:20.929176 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.902966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.1.188.194:35937)
0621 07:05:20.920543 (+ 17577us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.920551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.920579 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:05:20.920683 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.920692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.920913 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.921869 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:05:20.922165 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.923684 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.926049 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.927552 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.928383 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.928589 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.928659 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.928824 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17388,"thread_start_us":17233,"threads_started":1}
I20260621 07:05:20.929267 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.915156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33669 (local address 127.0.0.1:46648)
0621 07:05:20.918323 (+ 3167us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:20.918339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:20.918456 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:20.920967 (+ 2511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:20.920973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:20.920998 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:05:20.921497 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.921509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.923569 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.923574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.926199 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.926211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.928374 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.928382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.928565 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.928602 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:05:20.928701 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:05:20.928855 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3044,"mutex_wait_us":27,"thread_start_us":3352,"threads_started":1}
I20260621 07:05:20.929656 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.915600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.1.188.193:33669)
0621 07:05:20.920214 (+ 4614us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:20.920226 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:20.920258 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:05:20.920389 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:20.920397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:20.920638 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:20.920881 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:05:20.921761 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.923428 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.926356 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.927858 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.927996 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.928231 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:20.929073 (+ 842us) server_negotiation.cc:299] Negotiation successful
0621 07:05:20.929259 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4413,"thread_start_us":4194,"threads_started":1}
I20260621 07:05:20.929781 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:20.902066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37413 (local address 127.0.0.1:55028)
0621 07:05:20.915959 (+ 13893us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:20.916015 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:20.916269 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:20.920298 (+ 4029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:20.920307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:20.920339 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:05:20.920780 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.920798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.922745 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.922753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.925740 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:20.925761 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:20.927351 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:20.927357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:20.927578 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:20.927673 (+ 95us) client_negotiation.cc:769] Sending connection context
0621 07:05:20.927854 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 07:05:20.928077 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13510,"mutex_wait_us":291,"thread_start_us":15345,"threads_started":1}
I20260621 07:05:20.933979 12394 tablet_service.cc:1467] Processing CreateTablet for tablet f835b8994f884dfd89eb63a990d0dbdf (DEFAULT_TABLE table=test-table [id=2824f5f78a3948728e49aabc3c11ef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:05:20.934003 12528 tablet_service.cc:1467] Processing CreateTablet for tablet f835b8994f884dfd89eb63a990d0dbdf (DEFAULT_TABLE table=test-table [id=2824f5f78a3948728e49aabc3c11ef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:05:20.933995 12260 tablet_service.cc:1467] Processing CreateTablet for tablet f835b8994f884dfd89eb63a990d0dbdf (DEFAULT_TABLE table=test-table [id=2824f5f78a3948728e49aabc3c11ef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:05:20.935539 12394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f835b8994f884dfd89eb63a990d0dbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:20.935541 12260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f835b8994f884dfd89eb63a990d0dbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:20.935541 12528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f835b8994f884dfd89eb63a990d0dbdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:05:20.948930 12612 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Bootstrap starting.
I20260621 07:05:20.951594 12614 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Bootstrap starting.
I20260621 07:05:20.955737 12614 tablet_bootstrap.cc:654] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:20.955654 12612 tablet_bootstrap.cc:654] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:20.956372 12613 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Bootstrap starting.
I20260621 07:05:20.957433 12612 log.cc:824] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:20.957420 12614 log.cc:824] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:20.960265 12613 tablet_bootstrap.cc:654] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Neither blocks nor log segments found. Creating new log.
I20260621 07:05:20.961506 12613 log.cc:824] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Log is configured to *not* fsync() on all Append() calls
I20260621 07:05:20.961905 12614 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: No bootstrap required, opened a new log
I20260621 07:05:20.961916 12612 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: No bootstrap required, opened a new log
I20260621 07:05:20.962237 12612 ts_tablet_manager.cc:1397] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20260621 07:05:20.962239 12614 ts_tablet_manager.cc:1397] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260621 07:05:20.964712 12613 tablet_bootstrap.cc:492] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: No bootstrap required, opened a new log
I20260621 07:05:20.965021 12613 ts_tablet_manager.cc:1397] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 07:05:20.971102 12612 raft_consensus.cc:348] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.971635 12612 raft_consensus.cc:374] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:20.971769 12612 raft_consensus.cc:729] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bc8073d3834855aed4b18839cff667, State: Initialized, Role: FOLLOWER
I20260621 07:05:20.972538 12612 consensus_queue.cc:260] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.973907 12459 heartbeater.cc:503] Master 127.1.188.254:35805 was elected leader, sending a full tablet report...
I20260621 07:05:20.974680 12612 ts_tablet_manager.cc:1428] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260621 07:05:20.974566 12614 raft_consensus.cc:348] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.975035 12614 raft_consensus.cc:374] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:20.975145 12614 raft_consensus.cc:729] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a2f2077933422c9d74f643142dcee5, State: Initialized, Role: FOLLOWER
I20260621 07:05:20.975868 12614 consensus_queue.cc:260] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.976224 12613 raft_consensus.cc:348] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.976765 12613 raft_consensus.cc:374] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:05:20.976936 12613 raft_consensus.cc:729] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf4647e38fb4595abfae2ca8cc23c86, State: Initialized, Role: FOLLOWER
I20260621 07:05:20.977156 12593 heartbeater.cc:503] Master 127.1.188.254:35805 was elected leader, sending a full tablet report...
I20260621 07:05:20.977715 12613 consensus_queue.cc:260] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:20.977959 12614 ts_tablet_manager.cc:1428] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260621 07:05:20.979792 12613 ts_tablet_manager.cc:1428] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260621 07:05:20.988662 12326 tablet.cc:2369] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:05:21.020119 12618 raft_consensus.cc:482] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:05:21.020498 12618 raft_consensus.cc:504] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:21.022559 12618 leader_election.cc:288] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07a2f2077933422c9d74f643142dcee5 (127.1.188.195:37413), 6bf4647e38fb4595abfae2ca8cc23c86 (127.1.188.193:33669)
W20260621 07:05:21.025048 12460 tablet.cc:2369] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:05:21.031086 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.022950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:57855 (local address 127.1.188.195:37413)
0621 07:05:21.023213 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.023226 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.024080 (+ 854us) server_negotiation.cc:407] Connection header received
0621 07:05:21.024348 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.024352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.024443 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.024542 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:05:21.025353 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.026566 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.028922 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.030014 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.030206 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.030296 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.030615 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.030784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 07:05:21.031184 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.022865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:37413 (local address 127.1.188.194:57855)
0621 07:05:21.023959 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:21.023973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:21.024188 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:21.024566 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:21.024571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:21.024598 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:05:21.025117 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:21.025130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.026773 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.026776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:21.028751 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:21.028765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.030239 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.030245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:21.030424 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.030462 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:05:21.030556 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 07:05:21.030699 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":307,"spinlock_wait_cycles":41216,"thread_start_us":477,"threads_started":1}
I20260621 07:05:21.031952 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.023008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:33621 (local address 127.1.188.193:33669)
0621 07:05:21.023296 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.023306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.024038 (+ 732us) server_negotiation.cc:407] Connection header received
0621 07:05:21.024295 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.024300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.024368 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.024474 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:05:21.025412 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.026572 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.029452 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.030651 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.030812 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.030895 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.031523 (+ 628us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.031681 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:05:21.032073 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.022854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33669 (local address 127.1.188.194:33621)
0621 07:05:21.023905 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0621 07:05:21.023923 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:05:21.024129 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:05:21.024502 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:05:21.024506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:05:21.024530 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:05:21.025207 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:21.025220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.026711 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.026716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:21.029279 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:05:21.029293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.031074 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.031082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:05:21.031315 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.031347 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:05:21.031483 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 07:05:21.031627 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":103,"thread_start_us":607,"threads_started":1}
I20260621 07:05:21.032979 12548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f835b8994f884dfd89eb63a990d0dbdf" candidate_uuid: "d2bc8073d3834855aed4b18839cff667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a2f2077933422c9d74f643142dcee5" is_pre_election: true
I20260621 07:05:21.033319 12280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f835b8994f884dfd89eb63a990d0dbdf" candidate_uuid: "d2bc8073d3834855aed4b18839cff667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" is_pre_election: true
I20260621 07:05:21.033635 12548 raft_consensus.cc:2454] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2bc8073d3834855aed4b18839cff667 in term 0.
I20260621 07:05:21.033907 12280 raft_consensus.cc:2454] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2bc8073d3834855aed4b18839cff667 in term 0.
I20260621 07:05:21.034713 12348 leader_election.cc:302] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07a2f2077933422c9d74f643142dcee5, 6bf4647e38fb4595abfae2ca8cc23c86, d2bc8073d3834855aed4b18839cff667; no voters:
I20260621 07:05:21.035210 12618 raft_consensus.cc:2781] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:05:21.035368 12618 raft_consensus.cc:482] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:05:21.035503 12618 raft_consensus.cc:3037] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:21.038058 12618 raft_consensus.cc:504] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:21.038813 12618 leader_election.cc:288] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [CANDIDATE]: Term 1 election: Requested vote from peers 07a2f2077933422c9d74f643142dcee5 (127.1.188.195:37413), 6bf4647e38fb4595abfae2ca8cc23c86 (127.1.188.193:33669)
I20260621 07:05:21.039309 12548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f835b8994f884dfd89eb63a990d0dbdf" candidate_uuid: "d2bc8073d3834855aed4b18839cff667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07a2f2077933422c9d74f643142dcee5"
I20260621 07:05:21.039522 12280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f835b8994f884dfd89eb63a990d0dbdf" candidate_uuid: "d2bc8073d3834855aed4b18839cff667" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf4647e38fb4595abfae2ca8cc23c86"
I20260621 07:05:21.039589 12548 raft_consensus.cc:3037] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:21.039762 12280 raft_consensus.cc:3037] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:05:21.043203 12548 raft_consensus.cc:2454] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2bc8073d3834855aed4b18839cff667 in term 1.
I20260621 07:05:21.043232 12280 raft_consensus.cc:2454] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2bc8073d3834855aed4b18839cff667 in term 1.
I20260621 07:05:21.043746 12348 leader_election.cc:302] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07a2f2077933422c9d74f643142dcee5, 6bf4647e38fb4595abfae2ca8cc23c86, d2bc8073d3834855aed4b18839cff667; no voters:
I20260621 07:05:21.044116 12618 raft_consensus.cc:2781] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:05:21.044741 12618 raft_consensus.cc:686] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 1 LEADER]: Becoming Leader. State: Replica: d2bc8073d3834855aed4b18839cff667, State: Running, Role: LEADER
I20260621 07:05:21.045580 12618 consensus_queue.cc:237] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } }
I20260621 07:05:21.050786 12136 catalog_manager.cc:5495] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2bc8073d3834855aed4b18839cff667 (127.1.188.194). New cstate: current_term: 1 leader_uuid: "d2bc8073d3834855aed4b18839cff667" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bc8073d3834855aed4b18839cff667" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 } health_report { overall_health: UNKNOWN } } }
I20260621 07:05:21.070159 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.064711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.1.188.194:35937)
0621 07:05:21.064932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.064937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.064956 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:05:21.065144 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.065148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.065243 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.065389 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:05:21.065752 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.066511 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.067994 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.068393 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.068483 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.068925 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:05:21.069522 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:05:21.069636 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.069743 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.069813 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260621 07:05:21.073993 12594 tablet.cc:2369] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:05:21.079881 12280 raft_consensus.cc:1261] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 1 FOLLOWER]: Refusing update from remote peer d2bc8073d3834855aed4b18839cff667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:05:21.079881 12548 raft_consensus.cc:1261] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 1 FOLLOWER]: Refusing update from remote peer d2bc8073d3834855aed4b18839cff667: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:05:21.080958 12623 consensus_queue.cc:1035] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07a2f2077933422c9d74f643142dcee5" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 37413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:05:21.081364 12618 consensus_queue.cc:1035] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf4647e38fb4595abfae2ca8cc23c86" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:05:21.097792 12628 mvcc.cc:205] Tried to move back new op lower bound from 7299176534331543552 to 7299176534207160320. Current Snapshot: MvccSnapshot[applied={T|T < 7299176534331543552}]
I20260621 07:05:21.108728 12134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:05:21.109058 12134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:05:21.113484 12134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0690cb071df046e78bfaac42dfce8c55: Sending DeleteTablet for 3 replicas of tablet f835b8994f884dfd89eb63a990d0dbdf
I20260621 07:05:21.114984 12260 tablet_service.cc:1514] Processing DeleteTablet for tablet f835b8994f884dfd89eb63a990d0dbdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:05:21 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:46648
I20260621 07:05:21.114984 12394 tablet_service.cc:1514] Processing DeleteTablet for tablet f835b8994f884dfd89eb63a990d0dbdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:05:21 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:46604
I20260621 07:05:21.114969 12528 tablet_service.cc:1514] Processing DeleteTablet for tablet f835b8994f884dfd89eb63a990d0dbdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:05:21 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:55028
I20260621 07:05:21.117268 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 12197
I20260621 07:05:21.130447 12638 tablet_replica.cc:330] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: stopping tablet replica
I20260621 07:05:21.130831 12636 tablet_replica.cc:330] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: stopping tablet replica
I20260621 07:05:21.131906 12636 raft_consensus.cc:2229] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:05:21.132107 12638 raft_consensus.cc:2229] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:05:21.133517 12638 raft_consensus.cc:2258] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:05:21.133533 12636 raft_consensus.cc:2258] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:05:21.134809 12637 tablet_replica.cc:330] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: stopping tablet replica
I20260621 07:05:21.136727 12636 ts_tablet_manager.cc:1905] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:05:21.137059 12638 ts_tablet_manager.cc:1905] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:05:21.143095 12637 raft_consensus.cc:2229] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 1 LEADER]: Raft consensus shutting down.
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:05:21.144583 12637 raft_consensus.cc:2258] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:05:21.145635 12638 ts_tablet_manager.cc:1918] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:05:21.145882 12638 log.cc:1196] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/f835b8994f884dfd89eb63a990d0dbdf
I20260621 07:05:21.146975 12638 ts_tablet_manager.cc:1939] T f835b8994f884dfd89eb63a990d0dbdf P 07a2f2077933422c9d74f643142dcee5: Deleting consensus metadata
I20260621 07:05:21.147769 12636 ts_tablet_manager.cc:1918] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:05:21.148041 12636 log.cc:1196] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/f835b8994f884dfd89eb63a990d0dbdf
I20260621 07:05:21.148685 12636 ts_tablet_manager.cc:1939] T f835b8994f884dfd89eb63a990d0dbdf P 6bf4647e38fb4595abfae2ca8cc23c86: Deleting consensus metadata
I20260621 07:05:21.149847 12637 ts_tablet_manager.cc:1905] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:05:21.151938 12123 catalog_manager.cc:4841] TS 6bf4647e38fb4595abfae2ca8cc23c86 (127.1.188.193:33669): tablet f835b8994f884dfd89eb63a990d0dbdf (table test-table [id=2824f5f78a3948728e49aabc3c11ef0c]) successfully deleted
I20260621 07:05:21.152611 12121 catalog_manager.cc:4841] TS 07a2f2077933422c9d74f643142dcee5 (127.1.188.195:37413): tablet f835b8994f884dfd89eb63a990d0dbdf (table test-table [id=2824f5f78a3948728e49aabc3c11ef0c]) successfully deleted
I20260621 07:05:21.154541 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.119236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.1.188.193:33669)
0621 07:05:21.119537 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.119548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.132820 (+ 13272us) server_negotiation.cc:407] Connection header received
0621 07:05:21.133079 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.133085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.133200 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.133315 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:21.135343 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.136550 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.138345 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.139039 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.139134 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.145178 (+ 6044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:21.145199 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:21.145221 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:21.145272 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:21.150609 (+ 5337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.151810 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.151820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.151824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.152002 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.152541 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.152546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.152549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.153041 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:21.153287 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.153834 (+ 547us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.154089 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 07:05:21.155368 12320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:21.158157 12637 ts_tablet_manager.cc:1918] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:05:21.158427 12637 log.cc:1196] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/f835b8994f884dfd89eb63a990d0dbdf
I20260621 07:05:21.159117 12637 ts_tablet_manager.cc:1939] T f835b8994f884dfd89eb63a990d0dbdf P d2bc8073d3834855aed4b18839cff667: Deleting consensus metadata
I20260621 07:05:21.161235 12123 catalog_manager.cc:4841] TS d2bc8073d3834855aed4b18839cff667 (127.1.188.194:35937): tablet f835b8994f884dfd89eb63a990d0dbdf (table test-table [id=2824f5f78a3948728e49aabc3c11ef0c]) successfully deleted
I20260621 07:05:21.254383 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 12197
I20260621 07:05:21.276726 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 12331
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:05:21.296352 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.278070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.1.188.194:35937)
0621 07:05:21.278303 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.278317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.278336 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:05:21.278399 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.278403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.278475 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.278631 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:21.280290 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.281281 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.282697 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.283225 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.283472 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.288542 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:21.288559 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:21.288595 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:21.288660 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:21.292488 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.293290 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.293305 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.293309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.293494 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.293935 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.293943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.293947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.294415 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:21.294842 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.295175 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.295945 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 07:05:21.297298 12454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:21.425129 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 12331
I20260621 07:05:21.443869 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 12465
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:05:21.460975 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:05:21.444726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.1.188.195:37413)
0621 07:05:21.444939 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 07:05:21.444946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:05:21.445006 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:05:21.445244 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:05:21.445249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:05:21.445337 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:05:21.445453 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:05:21.446814 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.447726 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.449012 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:05:21.449457 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:05:21.449600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:05:21.454499 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:05:21.454515 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:05:21.454536 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:05:21.454581 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:05:21.457740 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.458489 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.458498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.458501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.458632 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:05:21.459019 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:05:21.459026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:05:21.459030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:05:21.459527 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:05:21.459849 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0621 07:05:21.460169 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:05:21.460422 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:05:21.461779 12588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:21.557569 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 12465
I20260621 07:05:21.576334 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 12104
I20260621 07:05:21.577605 12166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:05:21.703262 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 12104
2026-06-21T07:05:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4572 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12646](info): set up 2 sockets
Jun 21 07:05:21 dist-test-slave-hlq1 krb5kdc[12646](info): commencing operation
krb5kdc: starting...
W20260621 07:05:22.832638 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:05:22 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:05:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangeradmin_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangeradmin_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangerlookup_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangerlookup_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
I20260621 07:05:23.172731 1779 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRiL99G/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/postgres -l logfile start
2026-06-21 07:05:25.607 UTC [12676] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:05:25.607 UTC [12676] LOG: listening on IPv4 address "127.1.188.212", port 55019
2026-06-21 07:05:25.611 UTC [12676] LOG: listening on Unix socket "/tmp/.s.PGSQL.55019"
2026-06-21 07:05:25.638 UTC [12679] LOG: database system was shut down at 2026-06-21 07:05:24 UTC
2026-06-21 07:05:25.641 UTC [12676] LOG: database system is ready to accept connections
I20260621 07:05:26.628903 1779 mini_postgres.cc:96] Postgres bound to 55019
2026-06-21 07:05:26.651 UTC [12687] FATAL: database "slave" does not exist
127.1.188.212:55019 - accepting connections
I20260621 07:05:26.652616 1779 mini_ranger.cc:161] Starting Ranger...
I20260621 07:05:26.684012 1779 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:05:27.186977 1779 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:05:27.187124 1779 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin
2026-06-21 07:05:27,487 [I] DB FLAVOR :POSTGRES
2026-06-21 07:05:27,488 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:05:27,488 [I] Checking connection..
2026-06-21 07:05:27,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:05:27,761 [I] Checking connection passed.
2026-06-21 07:05:27,762 [I] --------- Verifying version history table ---------
2026-06-21 07:05:27,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:05:27,994 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:05:27,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:05:28,259 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:05:28,260 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:05:28,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:05:28.516 UTC [12751] WARNING: there is no transaction in progress
2026-06-21 07:05:28,523 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:05:28,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:05:28,782 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:05:28,782 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:05:28,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:05:29,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:05:29,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hlq1.c.gcp-upstream.internal','N') ;"
2026-06-21 07:05:29,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:05:29,763 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:05:29,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:05:30,004 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:05:30,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:05:30,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:05:30,508 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:05:30,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:05:31.428 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.443 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.459 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.473 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.488 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.505 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.559 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.566 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.579 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.589 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.599 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.613 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.625 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.632 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.638 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:31.645 UTC [12859] WARNING: there is no transaction in progress
2026-06-21 07:05:32,266 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:05:32,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:05:32,532 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:05:32,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:05:32,815 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:05:32,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:05:33,045 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:05:33,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:05:33,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:05:33,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hlq1.c.gcp-upstream.internal';"
2026-06-21 07:05:33,761 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:05:33,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:05:34,005 [I] DB_PATCHES have already been applied
I20260621 07:05:34.012176 1779 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRiL99G/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:05:34.012284 1779 mini_ranger.cc:193] Using host: 127.1.188.212
I20260621 07:05:34.028472 1779 mini_ranger.cc:227] Ranger admin URL: http://127.1.188.212:57285
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:05:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:05:35.167191 1779 mini_ranger.cc:160] Time spent starting Ranger: real 8.515s user 0.001s sys 0.047s
Jun 21, 2026 7:05:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 21, 2026 7:05:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:05:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:05:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:05:57.773506 1779 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangerkms_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/rangerkms_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.188.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.188.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:05:57 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025557, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:05:57.946201 1779 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:05:57.967248 1779 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:05:58.800127 1779 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:05:58.800343 1779 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms
I20260621 07:05:58.800519 1779 mini_ranger_kms.cc:227] Using postgres at 127.1.188.212:55019
2026-06-21 07:05:59,305 [I] DB FLAVOR :POSTGRES
2026-06-21 07:05:59,305 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:05:59,305 [I] Checking connection
2026-06-21 07:05:59,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:05:59,571 [I] connection success
2026-06-21 07:05:59,571 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:05:59,571 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:05:59,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:05:59,804 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:05:59,805 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:05:59,805 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:05:59,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:55019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:06:00,075 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:06:00.517419 1779 mini_ranger_kms.cc:310] Created kms service
I20260621 07:06:00.704224 1779 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:06:00.749993 1779 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:06:00 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025557, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:01.146708 1779 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:06:01.147034 1779 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:06:01.147079 1779 mini_ranger_kms.cc:240] Using host: 127.1.188.212
I20260621 07:06:01.174057 1779 mini_ranger_kms.cc:276] Ranger KMS PID: 13040
I20260621 07:06:01.174264 1779 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.188.212:38337
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:06:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 07:06:03.428398 1779 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.482s user 0.129s sys 0.393s
I20260621 07:06:03.428684 1779 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:06:03.428906 1779 mini_ranger_kms.cc:401] 127.1.188.212:38337/kms/v1/keys
Jun 21 07:06:03 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782025563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 07:06:04 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
Jun 21 07:06:04 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
Jun 21 07:06:06 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:06:06.741133 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41333
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41333
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:38337/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:06:07.095651 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:07.095923 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:07.096067 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:07.103228 13090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:06:07.103327 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:07.103446 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:07.103580 13090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:06:07.103662 13090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:06:07.113713 13090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.188.212:38337/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41333
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:07.114980 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:07.116369 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:07.128451 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:07.128505 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:07.128487 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:07.129204 13090 server_base.cc:1034] running on GCE node
I20260621 07:06:07.129995 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:07.131666 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:07.132917 13090 hybrid_clock.cc:648] HybridClock initialized: now 1782025567132866 us; error 46 us; skew 500 ppm
I20260621 07:06:07.133327 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:07 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:07.136718 13090 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:06:07.138198 13090 webserver.cc:460] Webserver started at http://127.1.188.254:33395/ using document root <none> and password file <none>
I20260621 07:06:07.138901 13090 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:07.139025 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:07.139354 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:07 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:07.194389 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "db04a16041f24bd1bbcf5ff45e3e01c9"
format_stamp: "Formatted at 2026-06-21 07:06:07 on dist-test-slave-hlq1"
server_key: "ad14f1c5b977e864211f01523fe62e65"
server_key_iv: "f8c15494410de36e21dd56539c637a7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:07.195227 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "db04a16041f24bd1bbcf5ff45e3e01c9"
format_stamp: "Formatted at 2026-06-21 07:06:07 on dist-test-slave-hlq1"
server_key: "ad14f1c5b977e864211f01523fe62e65"
server_key_iv: "f8c15494410de36e21dd56539c637a7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:07.200686 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 07:06:07.217096 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:07.218638 13090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:06:07.218842 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "db04a16041f24bd1bbcf5ff45e3e01c9"
format_stamp: "Formatted at 2026-06-21 07:06:07 on dist-test-slave-hlq1"
server_key: "ad14f1c5b977e864211f01523fe62e65"
server_key_iv: "f8c15494410de36e21dd56539c637a7a"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:07.219070 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:07.248654 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:07.252825 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:07.253147 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:07.273869 13090 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41333
I20260621 07:06:07.273957 13156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41333 every 8 connection(s)
I20260621 07:06:07.275269 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:06:07.279948 13157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:06:07.281257 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 13090
I20260621 07:06:07.281474 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:06:07.282176 1779 external_mini_cluster.cc:1428] Setting key 873edbef935dc24e0b352b7815cc044f
I20260621 07:06:07.294480 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: Bootstrap starting.
I20260621 07:06:07.298754 13157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: Neither blocks nor log segments found. Creating new log.
I20260621 07:06:07.300309 13157 log.cc:824] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:06:07.303843 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: No bootstrap required, opened a new log
Jun 21 07:06:07 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:06:07.314929 13157 raft_consensus.cc:348] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } }
I20260621 07:06:07.315521 13157 raft_consensus.cc:374] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:06:07.315644 13157 raft_consensus.cc:729] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db04a16041f24bd1bbcf5ff45e3e01c9, State: Initialized, Role: FOLLOWER
I20260621 07:06:07.316386 13157 consensus_queue.cc:260] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } }
I20260621 07:06:07.316675 13157 raft_consensus.cc:388] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:06:07.316856 13157 raft_consensus.cc:482] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:06:07.317029 13157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:06:07.318804 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:07.284494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.1.188.254:41333)
0621 07:06:07.299381 (+ 14887us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:07.299413 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:07.299469 (+ 56us) server_negotiation.cc:407] Connection header received
0621 07:06:07.301633 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:07.301666 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:07.302076 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:07.302436 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:07.303809 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:07.305182 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:07.306429 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:07.307105 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:07.307307 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:07.311203 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:07.311211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:07.311231 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:07.311269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:07.314203 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:07.314876 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:07.314883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:07.314888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:07.315056 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:07.315544 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:07.315548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:07.315550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:07.316213 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:07.316515 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:07.317511 (+ 996us) server_negotiation.cc:299] Negotiation successful
0621 07:06:07.317823 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14627,"thread_start_us":14342,"threads_started":1}
I20260621 07:06:07.320354 13157 raft_consensus.cc:504] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } }
I20260621 07:06:07.320930 13157 leader_election.cc:302] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db04a16041f24bd1bbcf5ff45e3e01c9; no voters:
I20260621 07:06:07.326601 13157 leader_election.cc:288] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:06:07.326678 13162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:06:07.328977 13157 sys_catalog.cc:564] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:06:07.333653 13162 raft_consensus.cc:686] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [term 1 LEADER]: Becoming Leader. State: Replica: db04a16041f24bd1bbcf5ff45e3e01c9, State: Running, Role: LEADER
I20260621 07:06:07.334187 13162 consensus_queue.cc:237] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } }
I20260621 07:06:07.337333 13157 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:06:07.337182 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } } }
I20260621 07:06:07.337599 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:06:07.337718 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db04a16041f24bd1bbcf5ff45e3e01c9. Latest consensus state: current_term: 1 leader_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db04a16041f24bd1bbcf5ff45e3e01c9" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41333 } } }
I20260621 07:06:07.338127 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:06:07.345016 13176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:06:07.352658 13176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:06:07.362797 13176 catalog_manager.cc:1347] Generated new cluster ID: 8d84322bc0ca4cb6a8c5913e0170a629
I20260621 07:06:07.362907 13176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:06:07.384855 13176 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:06:07.386876 13176 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:06:07.398547 13176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: Generated new TSK 0
I20260621 07:06:07.399714 13176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:06:08.115556 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:38337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:06:08 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:06:08.561094 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:08.561580 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:08.561741 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:08.573666 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:08.573823 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:08.574059 13202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:06:08.589848 13202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.188.212:38337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:08.591593 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:08.593293 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:08.624924 13213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:08.626175 13212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:08.629271 13202 server_base.cc:1034] running on GCE node
W20260621 07:06:08.629716 13215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:08.630676 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:08.632012 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:08.633330 13202 hybrid_clock.cc:648] HybridClock initialized: now 1782025568633245 us; error 71 us; skew 500 ppm
I20260621 07:06:08.633756 13202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:08 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:08.639272 13202 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:06:08.644925 13202 webserver.cc:460] Webserver started at http://127.1.188.193:39733/ using document root <none> and password file <none>
I20260621 07:06:08.645839 13202 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:08.645972 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:08.646341 13202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:08 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:08.665793 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "72604818fc2c4affb96a730ab242d568"
format_stamp: "Formatted at 2026-06-21 07:06:08 on dist-test-slave-hlq1"
server_key: "63470c2c4da66edb0c4657ed2e6fa71d"
server_key_iv: "768a2735da13d585b2835c338c05e325"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:08.666898 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "72604818fc2c4affb96a730ab242d568"
format_stamp: "Formatted at 2026-06-21 07:06:08 on dist-test-slave-hlq1"
server_key: "63470c2c4da66edb0c4657ed2e6fa71d"
server_key_iv: "768a2735da13d585b2835c338c05e325"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:08.672667 13202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 07:06:08.700464 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:08.702469 13202 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.003s
I20260621 07:06:08.702710 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "72604818fc2c4affb96a730ab242d568"
format_stamp: "Formatted at 2026-06-21 07:06:08 on dist-test-slave-hlq1"
server_key: "63470c2c4da66edb0c4657ed2e6fa71d"
server_key_iv: "768a2735da13d585b2835c338c05e325"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:08.702977 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:08.749372 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:08.754727 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:08.755060 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:08.762364 13202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:06:08.765583 13202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:06:08.765751 13202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 07:06:08.765960 13202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:06:08.766086 13202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:06:08 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:08.927780 13202 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:45195
I20260621 07:06:08.927798 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:45195 every 8 connection(s)
I20260621 07:06:08.929304 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:06:08.934598 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 13202
I20260621 07:06:08.934841 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:06:08.936062 1779 external_mini_cluster.cc:1428] Setting key 496d2606678c44f1266c7dc704458d37
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Jun 21 07:06:08 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:06:08.991389 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:08.943321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:34753 (local address 127.1.188.254:41333)
0621 07:06:08.961296 (+ 17975us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:08.961304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:08.961332 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:06:08.967784 (+ 6452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:08.967793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:08.967913 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:08.968040 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:08.970507 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:08.971821 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:08.974194 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:08.975024 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:08.975185 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:08.982553 (+ 7368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:08.982564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:08.982572 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:08.982624 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:08.986696 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:08.987900 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:08.987909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:08.987915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:08.988097 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:08.988780 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:08.988786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:08.988790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:08.989166 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:08.989316 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:08.990455 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0621 07:06:08.990790 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17812,"thread_start_us":18010,"threads_started":1}
I20260621 07:06:08.992524 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:08.942671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41333 (local address 127.1.188.193:34753)
0621 07:06:08.960373 (+ 17702us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:08.960467 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:08.963007 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:08.968513 (+ 5506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:08.968529 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:08.969360 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:06:08.970273 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:08.970304 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:08.972028 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:08.972034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:08.974016 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:08.974031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:08.975317 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:08.975325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:08.975674 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:08.976548 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:06:08.976580 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:06:08.982213 (+ 5633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:06:08.986931 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:08.986942 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:08.986963 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:08.987742 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:08.988258 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:08.988264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:08.988267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:08.988588 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:08.989353 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:06:08.989364 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:06:08.989933 (+ 569us) client_negotiation.cc:769] Sending connection context
0621 07:06:08.990279 (+ 346us) client_negotiation.cc:240] Negotiation successful
0621 07:06:08.990909 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17233,"thread_start_us":16934,"threads_started":1}
I20260621 07:06:08.994837 13335 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41333
I20260621 07:06:08.995191 13335 heartbeater.cc:464] Registering TS with master...
I20260621 07:06:08.996106 13335 heartbeater.cc:511] Master 127.1.188.254:41333 requested a full tablet report, sending...
I20260621 07:06:08.998672 13122 ts_manager.cc:194] Registered new tserver with Master: 72604818fc2c4affb96a730ab242d568 (127.1.188.193:45195)
I20260621 07:06:09.000963 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:34753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:06:09.015002 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:38337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:06:09 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
W20260621 07:06:09.446966 13341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:09.447532 13341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:09.447721 13341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:09.458827 13341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:09.459053 13341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:09.459296 13341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:06:09.474720 13341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.188.212:38337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:09.476648 13341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:09.478361 13341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:09.517241 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:09.517760 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:09.520394 13341 server_base.cc:1034] running on GCE node
W20260621 07:06:09.521220 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:09.522027 13341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:09.523290 13341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:09.524590 13341 hybrid_clock.cc:648] HybridClock initialized: now 1782025569524520 us; error 54 us; skew 500 ppm
I20260621 07:06:09.524942 13341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:09 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:09.529153 13341 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:06:09.533161 13341 webserver.cc:460] Webserver started at http://127.1.188.194:33431/ using document root <none> and password file <none>
I20260621 07:06:09.534243 13341 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:09.534406 13341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:09.534842 13341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:09 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:09.562376 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "85241e70a40944909286d65889a134dc"
format_stamp: "Formatted at 2026-06-21 07:06:09 on dist-test-slave-hlq1"
server_key: "9a016c3d0e4be532af46231968a865fa"
server_key_iv: "89e0574faef66271777f8865164b80f8"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:09.564057 13341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "85241e70a40944909286d65889a134dc"
format_stamp: "Formatted at 2026-06-21 07:06:09 on dist-test-slave-hlq1"
server_key: "9a016c3d0e4be532af46231968a865fa"
server_key_iv: "89e0574faef66271777f8865164b80f8"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:09.585541 13341 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.012s sys 0.004s
I20260621 07:06:09.623138 13365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:09.632642 13341 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.001s sys 0.007s
I20260621 07:06:09.632982 13341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "85241e70a40944909286d65889a134dc"
format_stamp: "Formatted at 2026-06-21 07:06:09 on dist-test-slave-hlq1"
server_key: "9a016c3d0e4be532af46231968a865fa"
server_key_iv: "89e0574faef66271777f8865164b80f8"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:09.633435 13341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:09.701973 13341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:09.707054 13341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:09.707386 13341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:09.708666 13341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:06:09.710462 13341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:06:09.710564 13341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:09.710739 13341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:06:09.710804 13341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:09.758008 13341 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:39977
I20260621 07:06:09.758050 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:39977 every 8 connection(s)
I20260621 07:06:09.759667 13341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:06:09.763144 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 13341
I20260621 07:06:09.763298 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:06:09.763852 1779 external_mini_cluster.cc:1428] Setting key b02b46172461cf18856c093342824fd0
Jun 21 07:06:09 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
I20260621 07:06:09.824656 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:09.763568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:40613 (local address 127.1.188.254:41333)
0621 07:06:09.796461 (+ 32893us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:09.796468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:09.800360 (+ 3892us) server_negotiation.cc:407] Connection header received
0621 07:06:09.800433 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:09.800438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:09.800534 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:09.800646 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:09.806912 (+ 6266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:09.808186 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:09.810642 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:09.811347 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:09.811489 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:09.816461 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:09.816469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:09.816475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:09.816519 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:09.820276 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:09.821354 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:09.821361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:09.821366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:09.821533 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:09.822180 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:09.822186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:09.822190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:09.822578 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:09.822725 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:09.823892 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0621 07:06:09.824210 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32738,"thread_start_us":32924,"threads_started":1}
I20260621 07:06:09.826957 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:09.762972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41333 (local address 127.1.188.194:40613)
0621 07:06:09.797218 (+ 34246us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:09.797288 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:09.799907 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:09.804546 (+ 4639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:09.804558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:09.805217 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:06:09.806693 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:09.806717 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:09.808397 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:09.808403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:09.810442 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:09.810457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:09.811851 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:09.811859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:09.812152 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:09.813158 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:06:09.813197 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:06:09.816184 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:06:09.820453 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:09.820463 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:09.820483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:09.821201 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:09.821698 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:09.821704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:09.821708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:09.822006 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:09.822842 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:06:09.822851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:06:09.823384 (+ 533us) client_negotiation.cc:769] Sending connection context
0621 07:06:09.824880 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0621 07:06:09.825442 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33886,"thread_start_us":40341,"threads_started":1}
I20260621 07:06:09.828889 13478 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41333
I20260621 07:06:09.829216 13478 heartbeater.cc:464] Registering TS with master...
I20260621 07:06:09.830035 13478 heartbeater.cc:511] Master 127.1.188.254:41333 requested a full tablet report, sending...
I20260621 07:06:09.832021 13122 ts_manager.cc:194] Registered new tserver with Master: 85241e70a40944909286d65889a134dc (127.1.188.194:39977)
I20260621 07:06:09.833247 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:40613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:06:09.860559 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:38337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:06:10.005210 13335 heartbeater.cc:503] Master 127.1.188.254:41333 was elected leader, sending a full tablet report...
W20260621 07:06:10.233444 13484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:10.233743 13484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:10.233835 13484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:10.241101 13484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:10.241206 13484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:10.241315 13484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:06:10.250774 13484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.188.212:38337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:10.251991 13484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:10.253157 13484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:10.265560 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:10.265671 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:10.265813 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:10.266700 13484 server_base.cc:1034] running on GCE node
I20260621 07:06:10.267256 13484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:10.268235 13484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:10.269485 13484 hybrid_clock.cc:648] HybridClock initialized: now 1782025570269439 us; error 43 us; skew 500 ppm
I20260621 07:06:10.269779 13484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:10 dist-test-slave-hlq1 krb5kdc[12646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:10.272892 13484 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:06:10.274497 13484 webserver.cc:460] Webserver started at http://127.1.188.195:43933/ using document root <none> and password file <none>
I20260621 07:06:10.275271 13484 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:10.275388 13484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:10.275740 13484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:10 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:10.289786 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "75f2d3ddc5d1483f8631baf980392746"
format_stamp: "Formatted at 2026-06-21 07:06:10 on dist-test-slave-hlq1"
server_key: "337ec70c983e7ec0d96a61e6c8bed4b9"
server_key_iv: "4530d560a8e8ccb84e0e99501d44e0bc"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:10.290628 13484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "75f2d3ddc5d1483f8631baf980392746"
format_stamp: "Formatted at 2026-06-21 07:06:10 on dist-test-slave-hlq1"
server_key: "337ec70c983e7ec0d96a61e6c8bed4b9"
server_key_iv: "4530d560a8e8ccb84e0e99501d44e0bc"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:10.296070 13484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 07:06:10.313406 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:10.315176 13484 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260621 07:06:10.315357 13484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "75f2d3ddc5d1483f8631baf980392746"
format_stamp: "Formatted at 2026-06-21 07:06:10 on dist-test-slave-hlq1"
server_key: "337ec70c983e7ec0d96a61e6c8bed4b9"
server_key_iv: "4530d560a8e8ccb84e0e99501d44e0bc"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:10.315660 13484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:10.340842 13484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:10.344703 13484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:10.344960 13484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:10.345841 13484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:06:10.347147 13484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:06:10.347218 13484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:10.347301 13484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:06:10.347343 13484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:10.380760 13484 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:44923
I20260621 07:06:10.380797 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:44923 every 8 connection(s)
I20260621 07:06:10.382057 13484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:06:10.389768 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 13484
I20260621 07:06:10.389950 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:06:10.390462 1779 external_mini_cluster.cc:1428] Setting key 1954ed26b21454eaf3404bcce294fe93
Jun 21 07:06:10 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025570, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:06:10.401459 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.384596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:49719 (local address 127.1.188.254:41333)
0621 07:06:10.385105 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.385110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.385863 (+ 753us) server_negotiation.cc:407] Connection header received
0621 07:06:10.387634 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.387640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.387729 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.387853 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:10.389200 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.390278 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.391938 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.392559 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.392680 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.396077 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:10.396084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:10.396089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:10.396130 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:10.398552 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.399221 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.399227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.399230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.399404 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.399839 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.399844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.399847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.400110 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:10.400259 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.400899 (+ 640us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.401105 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260621 07:06:10.402370 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.385052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41333 (local address 127.1.188.195:49719)
0621 07:06:10.385708 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.385745 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.387314 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.388074 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.388082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.388498 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:06:10.389032 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.389048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.390438 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.390442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.391795 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.391806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.392783 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.392790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.393027 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.393791 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:06:10.393827 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:06:10.395867 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:06:10.398686 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:10.398694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:10.398711 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:10.399101 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:10.399540 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:06:10.399546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:06:10.399550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:06:10.399720 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:06:10.400233 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:06:10.400240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:06:10.400557 (+ 317us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.400807 (+ 250us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.401287 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260621 07:06:10.403795 13612 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41333
I20260621 07:06:10.404111 13612 heartbeater.cc:464] Registering TS with master...
I20260621 07:06:10.404937 13612 heartbeater.cc:511] Master 127.1.188.254:41333 requested a full tablet report, sending...
I20260621 07:06:10.406409 13122 ts_manager.cc:194] Registered new tserver with Master: 75f2d3ddc5d1483f8631baf980392746 (127.1.188.195:44923)
I20260621 07:06:10.407240 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:49719
I20260621 07:06:10.418010 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:06:10.435111 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.422862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.1.188.254:41333)
0621 07:06:10.423044 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.423048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.423558 (+ 510us) server_negotiation.cc:407] Connection header received
0621 07:06:10.423690 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.423697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.423773 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.423863 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:10.425312 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.426018 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.427274 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.427756 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.427857 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.430184 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:10.430189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:10.430193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:10.430222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:10.432962 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.433550 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.433554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.433556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.433706 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.434009 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.434012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.434014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.434284 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:10.434462 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.434738 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.434894 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:06:10.438939 13122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:06:10.445921 13122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:06:10.469710 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.458580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.1.188.195:44923)
0621 07:06:10.462210 (+ 3630us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.462216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.462240 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:06:10.462311 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.462316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.462504 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.462706 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:06:10.463430 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.464792 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.467071 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.468484 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.468643 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.468944 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.469186 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.469346 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3501,"thread_start_us":3339,"threads_started":1}
I20260621 07:06:10.469707 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.458373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44923 (local address 127.0.0.1:47386)
0621 07:06:10.460783 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.460802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.460911 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.462710 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.462717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.462745 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:06:10.463227 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.463240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.464911 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.464919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.466922 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.466933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.468710 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.468717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.468940 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.469003 (+ 63us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.469144 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.469339 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2282,"thread_start_us":331,"threads_started":1}
I20260621 07:06:10.472325 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.460239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45195 (local address 127.0.0.1:47396)
0621 07:06:10.463761 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.463785 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.464137 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.464971 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.464975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.465002 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:06:10.465738 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.465760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.467705 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.467712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.470043 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.470054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.471485 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.471492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.471653 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.471700 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.471801 (+ 101us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.471929 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3378,"mutex_wait_us":1,"thread_start_us":825,"threads_started":1}
I20260621 07:06:10.472384 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.460885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.1.188.193:45195)
0621 07:06:10.461843 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.461854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.463894 (+ 2040us) server_negotiation.cc:407] Connection header received
0621 07:06:10.464333 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.464340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.464564 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.464733 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:06:10.465960 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.467531 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.470161 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.471251 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.471417 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.471650 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.471854 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.472018 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":646,"threads_started":1}
I20260621 07:06:10.473546 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.458211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39977 (local address 127.0.0.1:48690)
0621 07:06:10.459054 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.459089 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.459261 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.464925 (+ 5664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.464930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.464951 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:06:10.465455 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.465521 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.467555 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.467560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.469891 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.469910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.472768 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.472774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.472947 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.472978 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.473076 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.473203 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":30,"thread_start_us":5267,"threads_started":1}
I20260621 07:06:10.473639 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.458375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.1.188.194:39977)
0621 07:06:10.464330 (+ 5955us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.464339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.464365 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:06:10.464474 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.464481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.464707 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.464941 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:06:10.465732 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.467374 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.470058 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.472571 (+ 2513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.472698 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.472890 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.473133 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.473300 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4832,"thread_start_us":4723,"threads_started":1}
I20260621 07:06:10.474365 13547 tablet_service.cc:1467] Processing CreateTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 (DEFAULT_TABLE table=test-table [id=4a74b617b6dc45dc8e118f990ba4d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:06:10.475653 13547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b05f6174f94b738eaa9c774aa47b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:06:10.475724 13270 tablet_service.cc:1467] Processing CreateTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 (DEFAULT_TABLE table=test-table [id=4a74b617b6dc45dc8e118f990ba4d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:06:10.477053 13270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b05f6174f94b738eaa9c774aa47b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:06:10.482911 13413 tablet_service.cc:1467] Processing CreateTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 (DEFAULT_TABLE table=test-table [id=4a74b617b6dc45dc8e118f990ba4d222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:06:10.484345 13413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b05f6174f94b738eaa9c774aa47b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:06:10.487877 13631 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Bootstrap starting.
I20260621 07:06:10.489941 13632 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Bootstrap starting.
I20260621 07:06:10.492292 13631 tablet_bootstrap.cc:654] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Neither blocks nor log segments found. Creating new log.
I20260621 07:06:10.493533 13632 tablet_bootstrap.cc:654] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Neither blocks nor log segments found. Creating new log.
I20260621 07:06:10.493675 13631 log.cc:824] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Log is configured to *not* fsync() on all Append() calls
I20260621 07:06:10.494741 13632 log.cc:824] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Log is configured to *not* fsync() on all Append() calls
I20260621 07:06:10.497025 13631 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: No bootstrap required, opened a new log
I20260621 07:06:10.497328 13631 ts_tablet_manager.cc:1397] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 07:06:10.497664 13632 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: No bootstrap required, opened a new log
I20260621 07:06:10.497932 13632 ts_tablet_manager.cc:1397] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 07:06:10.498474 13635 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Bootstrap starting.
I20260621 07:06:10.502043 13635 tablet_bootstrap.cc:654] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Neither blocks nor log segments found. Creating new log.
I20260621 07:06:10.503183 13635 log.cc:824] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:06:10.506846 13635 tablet_bootstrap.cc:492] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: No bootstrap required, opened a new log
I20260621 07:06:10.507112 13635 ts_tablet_manager.cc:1397] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 07:06:10.508086 13631 raft_consensus.cc:348] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.508662 13631 raft_consensus.cc:374] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:06:10.508780 13631 raft_consensus.cc:729] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72604818fc2c4affb96a730ab242d568, State: Initialized, Role: FOLLOWER
I20260621 07:06:10.508764 13632 raft_consensus.cc:348] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.509227 13632 raft_consensus.cc:374] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:06:10.509331 13632 raft_consensus.cc:729] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f2d3ddc5d1483f8631baf980392746, State: Initialized, Role: FOLLOWER
I20260621 07:06:10.509406 13631 consensus_queue.cc:260] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.509893 13632 consensus_queue.cc:260] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.511461 13631 ts_tablet_manager.cc:1428] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260621 07:06:10.512562 13612 heartbeater.cc:503] Master 127.1.188.254:41333 was elected leader, sending a full tablet report...
I20260621 07:06:10.513140 13632 ts_tablet_manager.cc:1428] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260621 07:06:10.517081 13635 raft_consensus.cc:348] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.517505 13635 raft_consensus.cc:374] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:06:10.517622 13635 raft_consensus.cc:729] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85241e70a40944909286d65889a134dc, State: Initialized, Role: FOLLOWER
I20260621 07:06:10.518180 13635 consensus_queue.cc:260] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.519114 13478 heartbeater.cc:503] Master 127.1.188.254:41333 was elected leader, sending a full tablet report...
I20260621 07:06:10.519727 13635 ts_tablet_manager.cc:1428] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
W20260621 07:06:10.558384 13479 tablet.cc:2369] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:06:10.633733 13613 tablet.cc:2369] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:06:10.641007 13637 raft_consensus.cc:482] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:06:10.641330 13637 raft_consensus.cc:504] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.643054 13637 leader_election.cc:288] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85241e70a40944909286d65889a134dc (127.1.188.194:39977), 75f2d3ddc5d1483f8631baf980392746 (127.1.188.195:44923)
I20260621 07:06:10.649477 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.643605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:53291 (local address 127.1.188.195:44923)
0621 07:06:10.643873 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.643882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.644275 (+ 393us) server_negotiation.cc:407] Connection header received
0621 07:06:10.644517 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.644521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.644579 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.644676 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:06:10.645194 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.645899 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.647767 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.648536 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.648743 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.648818 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.649092 (+ 274us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.649215 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 07:06:10.649616 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.643428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:37035 (local address 127.1.188.194:39977)
0621 07:06:10.643655 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.643664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.644568 (+ 904us) server_negotiation.cc:407] Connection header received
0621 07:06:10.644771 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.644774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.644831 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.644920 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:06:10.645476 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.646163 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.647903 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.648663 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.648834 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.648912 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.649204 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.649322 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:06:10.649653 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.643469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:39977 (local address 127.1.188.193:37035)
0621 07:06:10.644467 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.644481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.644630 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.644953 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.644957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.644988 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:06:10.645333 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.645341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.646298 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.646303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.647759 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.647771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.648886 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.648892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.649029 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.649058 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.649165 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.649283 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":670,"threads_started":1}
I20260621 07:06:10.649700 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.643337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44923 (local address 127.1.188.193:53291)
0621 07:06:10.644159 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0621 07:06:10.644174 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:06:10.644370 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:06:10.644691 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:06:10.644695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:06:10.644720 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:06:10.645036 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.645048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.646041 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.646048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.647622 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:06:10.647634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.648756 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.648762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:06:10.648937 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.648961 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:06:10.649067 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:06:10.649204 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":4,"thread_start_us":512,"threads_started":1}
I20260621 07:06:10.650720 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b05f6174f94b738eaa9c774aa47b93" candidate_uuid: "72604818fc2c4affb96a730ab242d568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f2d3ddc5d1483f8631baf980392746" is_pre_election: true
I20260621 07:06:10.650722 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b05f6174f94b738eaa9c774aa47b93" candidate_uuid: "72604818fc2c4affb96a730ab242d568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85241e70a40944909286d65889a134dc" is_pre_election: true
I20260621 07:06:10.651284 13567 raft_consensus.cc:2454] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72604818fc2c4affb96a730ab242d568 in term 0.
I20260621 07:06:10.651284 13433 raft_consensus.cc:2454] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72604818fc2c4affb96a730ab242d568 in term 0.
I20260621 07:06:10.651966 13225 leader_election.cc:302] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72604818fc2c4affb96a730ab242d568, 75f2d3ddc5d1483f8631baf980392746; no voters:
I20260621 07:06:10.652359 13637 raft_consensus.cc:2781] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:06:10.652515 13637 raft_consensus.cc:482] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:06:10.652626 13637 raft_consensus.cc:3037] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:06:10.654815 13637 raft_consensus.cc:504] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.655470 13637 leader_election.cc:288] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [CANDIDATE]: Term 1 election: Requested vote from peers 85241e70a40944909286d65889a134dc (127.1.188.194:39977), 75f2d3ddc5d1483f8631baf980392746 (127.1.188.195:44923)
I20260621 07:06:10.655905 13567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b05f6174f94b738eaa9c774aa47b93" candidate_uuid: "72604818fc2c4affb96a730ab242d568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f2d3ddc5d1483f8631baf980392746"
I20260621 07:06:10.655913 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b05f6174f94b738eaa9c774aa47b93" candidate_uuid: "72604818fc2c4affb96a730ab242d568" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85241e70a40944909286d65889a134dc"
I20260621 07:06:10.656103 13567 raft_consensus.cc:3037] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:06:10.656173 13433 raft_consensus.cc:3037] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:06:10.658236 13567 raft_consensus.cc:2454] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72604818fc2c4affb96a730ab242d568 in term 1.
I20260621 07:06:10.658313 13433 raft_consensus.cc:2454] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72604818fc2c4affb96a730ab242d568 in term 1.
I20260621 07:06:10.658667 13225 leader_election.cc:302] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72604818fc2c4affb96a730ab242d568, 75f2d3ddc5d1483f8631baf980392746; no voters:
I20260621 07:06:10.658987 13637 raft_consensus.cc:2781] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:06:10.659381 13637 raft_consensus.cc:686] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 1 LEADER]: Becoming Leader. State: Replica: 72604818fc2c4affb96a730ab242d568, State: Running, Role: LEADER
I20260621 07:06:10.659974 13637 consensus_queue.cc:237] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } }
I20260621 07:06:10.664443 13122 catalog_manager.cc:5495] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72604818fc2c4affb96a730ab242d568 (127.1.188.193). New cstate: current_term: 1 leader_uuid: "72604818fc2c4affb96a730ab242d568" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72604818fc2c4affb96a730ab242d568" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45195 } health_report { overall_health: HEALTHY } } }
I20260621 07:06:10.713526 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.707117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.1.188.193:45195)
0621 07:06:10.707319 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.707324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.707364 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:06:10.707666 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.707670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.707779 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.707940 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:06:10.708513 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.709406 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.711002 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.711580 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.711688 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.712240 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:06:10.712950 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:06:10.713075 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.713199 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.713278 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 07:06:10.719681 13336 tablet.cc:2369] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:06:10.723716 13433 raft_consensus.cc:1261] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 1 FOLLOWER]: Refusing update from remote peer 72604818fc2c4affb96a730ab242d568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:06:10.723716 13567 raft_consensus.cc:1261] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 1 FOLLOWER]: Refusing update from remote peer 72604818fc2c4affb96a730ab242d568: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:06:10.724802 13642 consensus_queue.cc:1035] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f2d3ddc5d1483f8631baf980392746" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:06:10.725170 13637 consensus_queue.cc:1035] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85241e70a40944909286d65889a134dc" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:06:10.740530 13646 mvcc.cc:205] Tried to move back new op lower bound from 7299176737672044544 to 7299176737427570688. Current Snapshot: MvccSnapshot[applied={T|T < 7299176737672044544}]
I20260621 07:06:10.756083 13121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:06:10.756440 13121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:06:10.760542 13121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db04a16041f24bd1bbcf5ff45e3e01c9: Sending DeleteTablet for 3 replicas of tablet 92b05f6174f94b738eaa9c774aa47b93
I20260621 07:06:10.761759 13413 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:06:10 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:48690
I20260621 07:06:10.761955 13270 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:06:10 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:47396
I20260621 07:06:10.762645 13547 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b05f6174f94b738eaa9c774aa47b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:06:10 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:47386
I20260621 07:06:10.769150 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 13202
I20260621 07:06:10.772859 13656 tablet_replica.cc:330] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: stopping tablet replica
I20260621 07:06:10.773051 13654 tablet_replica.cc:330] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: stopping tablet replica
I20260621 07:06:10.773552 13654 raft_consensus.cc:2229] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:06:10.773552 13656 raft_consensus.cc:2229] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:06:10.774139 13654 raft_consensus.cc:2258] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:06:10.774139 13656 raft_consensus.cc:2258] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:06:10.776692 13656 ts_tablet_manager.cc:1905] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:06:10.784269 13656 ts_tablet_manager.cc:1918] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:06:10.784319 13655 tablet_replica.cc:330] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: stopping tablet replica
I20260621 07:06:10.784868 13655 raft_consensus.cc:2229] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:06:10.785854 13655 raft_consensus.cc:2258] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:06:10.785951 13654 ts_tablet_manager.cc:1905] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:06:10.790757 13655 ts_tablet_manager.cc:1905] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:06:10.793561 13654 ts_tablet_manager.cc:1918] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:06:10.793823 13656 log.cc:1196] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/92b05f6174f94b738eaa9c774aa47b93
I20260621 07:06:10.794037 13654 log.cc:1196] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/92b05f6174f94b738eaa9c774aa47b93
I20260621 07:06:10.794687 13656 ts_tablet_manager.cc:1939] T 92b05f6174f94b738eaa9c774aa47b93 P 75f2d3ddc5d1483f8631baf980392746: Deleting consensus metadata
I20260621 07:06:10.795353 13654 ts_tablet_manager.cc:1939] T 92b05f6174f94b738eaa9c774aa47b93 P 85241e70a40944909286d65889a134dc: Deleting consensus metadata
I20260621 07:06:10.797183 13108 catalog_manager.cc:4841] TS 75f2d3ddc5d1483f8631baf980392746 (127.1.188.195:44923): tablet 92b05f6174f94b738eaa9c774aa47b93 (table test-table [id=4a74b617b6dc45dc8e118f990ba4d222]) successfully deleted
I20260621 07:06:10.797199 13106 catalog_manager.cc:4841] TS 85241e70a40944909286d65889a134dc (127.1.188.194:39977): tablet 92b05f6174f94b738eaa9c774aa47b93 (table test-table [id=4a74b617b6dc45dc8e118f990ba4d222]) successfully deleted
Jun 21 07:06:10 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:06:10.801213 13655 ts_tablet_manager.cc:1918] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:06:10.801455 13655 log.cc:1196] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/92b05f6174f94b738eaa9c774aa47b93
I20260621 07:06:10.802150 13655 ts_tablet_manager.cc:1939] T 92b05f6174f94b738eaa9c774aa47b93 P 72604818fc2c4affb96a730ab242d568: Deleting consensus metadata
I20260621 07:06:10.804360 13107 catalog_manager.cc:4841] TS 72604818fc2c4affb96a730ab242d568 (127.1.188.193:45195): tablet 92b05f6174f94b738eaa9c774aa47b93 (table test-table [id=4a74b617b6dc45dc8e118f990ba4d222]) successfully deleted
I20260621 07:06:10.808497 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.771216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.1.188.193:45195)
0621 07:06:10.771842 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.771850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.780632 (+ 8782us) server_negotiation.cc:407] Connection header received
0621 07:06:10.780894 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.780902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.781011 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.781142 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:10.783422 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.785034 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.791015 (+ 5981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.791837 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.791960 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.801512 (+ 9552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:10.801531 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:10.801555 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:10.801620 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:10.805281 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.806069 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.806076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.806079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.806229 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.806640 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.806649 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.806651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.807045 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:10.807370 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.807810 (+ 440us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.808062 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260621 07:06:10.809406 13330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:06:10.919199 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 13202
I20260621 07:06:10.940927 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 13341
Jun 21 07:06:10 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:06:10.957279 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:10.941951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.1.188.194:39977)
0621 07:06:10.942148 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:10.942153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:10.942172 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:06:10.942326 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:10.942330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:10.942407 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:10.942495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:10.944155 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.944952 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.946427 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:10.946843 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:10.946942 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:10.951719 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:10.951735 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:10.951755 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:10.951793 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:10.954604 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.955241 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.955248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.955250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.955373 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:10.955809 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:10.955812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:10.955814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:10.956211 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:10.956446 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:10.956710 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:06:10.956876 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:06:10.958087 13473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:06:11.055953 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 13341
I20260621 07:06:11.075125 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 13484
Jun 21 07:06:11 dist-test-slave-hlq1 krb5kdc[12646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:06:11.092754 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:11.076088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.1.188.195:44923)
0621 07:06:11.076292 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:11.076299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:11.076377 (+ 78us) server_negotiation.cc:407] Connection header received
0621 07:06:11.076636 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:11.076642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:11.076732 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:11.076858 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:11.078277 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:11.079065 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:11.080586 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:11.080999 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:11.081138 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:11.086135 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:11.086157 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:11.086178 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:11.086219 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:11.089418 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:11.090188 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:11.090197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:11.090200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:11.090364 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:11.090794 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:11.090802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:11.090805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:11.091262 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:11.091708 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:11.092072 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 07:06:11.092319 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:06:11.093554 13607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:06:11.189442 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 13484
I20260621 07:06:11.207861 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 13090
I20260621 07:06:11.209241 13152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:06:11.335201 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 13090
I20260621 07:06:11.361732 1779 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:06:11.697260 1779 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:06:11.697567 1779 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:06:12.043612 1779 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:06:12.043 UTC [12676] LOG: received smart shutdown request
2026-06-21 07:06:12.047 UTC [12676] LOG: background worker "logical replication launcher" (PID 12685) exited with exit code 1
2026-06-21 07:06:12.047 UTC [12680] LOG: shutting down
2026-06-21 07:06:12.197 UTC [12676] LOG: database system is shut down
2026-06-21T07:06:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50628 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:06:12 dist-test-slave-hlq1 krb5kdc[13670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:06:12 dist-test-slave-hlq1 krb5kdc[13670](info): set up 2 sockets
Jun 21 07:06:12 dist-test-slave-hlq1 krb5kdc[13670](info): commencing operation
krb5kdc: starting...
W20260621 07:06:13.463937 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:06:13 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:06:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangeradmin_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangeradmin_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangerlookup_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangerlookup_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
I20260621 07:06:13.783627 1779 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRiL99G/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/postgres -l logfile start
2026-06-21 07:06:16.263 UTC [13700] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 07:06:16.263 UTC [13700] LOG: listening on IPv4 address "127.1.188.212", port 44367
2026-06-21 07:06:16.267 UTC [13700] LOG: listening on Unix socket "/tmp/.s.PGSQL.44367"
2026-06-21 07:06:16.292 UTC [13703] LOG: database system was shut down at 2026-06-21 07:06:14 UTC
2026-06-21 07:06:16.295 UTC [13700] LOG: database system is ready to accept connections
I20260621 07:06:17.315861 1779 mini_postgres.cc:96] Postgres bound to 44367
2026-06-21 07:06:17.336 UTC [13711] FATAL: database "slave" does not exist
127.1.188.212:44367 - accepting connections
I20260621 07:06:17.337491 1779 mini_ranger.cc:161] Starting Ranger...
I20260621 07:06:17.367774 1779 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 07:06:17.838707 1779 mini_ranger.cc:87] Created ranger Postgres database
I20260621 07:06:17.838866 1779 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin
2026-06-21 07:06:18,152 [I] DB FLAVOR :POSTGRES
2026-06-21 07:06:18,153 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:06:18,153 [I] Checking connection..
2026-06-21 07:06:18,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 07:06:18,425 [I] Checking connection passed.
2026-06-21 07:06:18,426 [I] --------- Verifying version history table ---------
2026-06-21 07:06:18,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:06:18,711 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:06:18,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:06:18,964 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 07:06:18,965 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 07:06:18,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 07:06:19.222 UTC [13775] WARNING: there is no transaction in progress
2026-06-21 07:06:19,229 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 07:06:19,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 07:06:19,484 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 07:06:19,484 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:06:19,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 07:06:19,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 07:06:20,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hlq1.c.gcp-upstream.internal','N') ;"
2026-06-21 07:06:20,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:06:20,508 [I] Table x_portal_user does not exist in database ranger
2026-06-21 07:06:20,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:06:20,754 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 07:06:20,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:06:21,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:06:21,301 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 07:06:21,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 07:06:22.290 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.308 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.325 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.340 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.359 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.380 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.441 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.447 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.461 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.471 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.480 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.494 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.504 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.510 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.517 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:22.523 UTC [13883] WARNING: there is no transaction in progress
2026-06-21 07:06:23,138 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 07:06:23,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 07:06:23,416 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 07:06:23,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 07:06:23,700 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 07:06:23,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 07:06:23,975 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 07:06:23,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 07:06:24,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 07:06:24,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hlq1.c.gcp-upstream.internal';"
2026-06-21 07:06:24,761 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 07:06:24,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 07:06:25,037 [I] DB_PATCHES have already been applied
I20260621 07:06:25.044663 1779 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRiL99G/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 07:06:25.044806 1779 mini_ranger.cc:193] Using host: 127.1.188.212
I20260621 07:06:25.063057 1779 mini_ranger.cc:227] Ranger admin URL: http://127.1.188.212:36187
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 7:06:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 07:06:26.247926 1779 mini_ranger.cc:160] Time spent starting Ranger: real 8.910s user 0.009s sys 0.042s
Jun 21, 2026 7:06:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 7:06:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 21, 2026 7:06:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 7:06:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 7:06:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 07:06:48.356966 1779 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.188.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.188.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangerkms_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.188.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/rangerkms_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.188.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.188.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.188.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/HTTP_127.1.188.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 07:06:48 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025608, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 07:06:48.572494 1779 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 07:06:48.600219 1779 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 07:06:49.432286 1779 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 07:06:49.432396 1779 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms
I20260621 07:06:49.432442 1779 mini_ranger_kms.cc:227] Using postgres at 127.1.188.212:44367
2026-06-21 07:06:49,787 [I] DB FLAVOR :POSTGRES
2026-06-21 07:06:49,787 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 07:06:49,787 [I] Checking connection
2026-06-21 07:06:49,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 07:06:50,025 [I] connection success
2026-06-21 07:06:50,025 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 07:06:50,025 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 07:06:50,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 07:06:50,262 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 07:06:50,262 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 07:06:50,263 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 07:06:50,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.188.212:44367/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 07:06:50,528 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 07:06:51.029779 1779 mini_ranger_kms.cc:310] Created kms service
I20260621 07:06:51.228030 1779 mini_ranger_kms.cc:326] Created kudu user
I20260621 07:06:51.273156 1779 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 07:06:51 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025608, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:51.659854 1779 mini_ranger_kms.cc:384] Added ranger policy
I20260621 07:06:51.660169 1779 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRiL99G/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRiL99G/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 07:06:51.660213 1779 mini_ranger_kms.cc:240] Using host: 127.1.188.212
I20260621 07:06:51.679307 1779 mini_ranger_kms.cc:276] Ranger KMS PID: 14063
I20260621 07:06:51.679461 1779 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.188.212:44075
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 7:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 07:06:54.038228 1779 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.466s user 0.077s sys 0.296s
I20260621 07:06:54.038491 1779 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 07:06:54.038762 1779 mini_ranger_kms.cc:401] 127.1.188.212:44075/kms/v1/keys
Jun 21 07:06:54 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782025614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 07:06:54 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
Jun 21 07:06:55 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025614, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.188.212@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
Jun 21 07:06:57 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:06:57.505509 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40357
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40357
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:06:57.840059 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:57.840399 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:57.840567 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:57.847920 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:06:57.848064 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:57.848184 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:57.848289 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:06:57.848371 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:06:57.858865 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.188.212:44075/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40357
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:57.860179 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:57.861467 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:57.874106 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:57.874045 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:57.874017 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:57.874821 14114 server_base.cc:1034] running on GCE node
I20260621 07:06:57.875723 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:57.877197 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:57.878444 14114 hybrid_clock.cc:648] HybridClock initialized: now 1782025617878395 us; error 56 us; skew 500 ppm
I20260621 07:06:57.878881 14114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:57 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:57.882098 14114 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:06:57.883811 14114 webserver.cc:460] Webserver started at http://127.1.188.254:37703/ using document root <none> and password file <none>
I20260621 07:06:57.884475 14114 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:57.884613 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:57.884918 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:57 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:57.937502 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "e71f8245c0194be19ad6e098599d33e6"
format_stamp: "Formatted at 2026-06-21 07:06:57 on dist-test-slave-hlq1"
server_key: "8f802827c4d38330c34e21ab6886699f"
server_key_iv: "b3d2aa84bd8f81899d6e2c923e3d165f"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:57.938611 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "e71f8245c0194be19ad6e098599d33e6"
format_stamp: "Formatted at 2026-06-21 07:06:57 on dist-test-slave-hlq1"
server_key: "8f802827c4d38330c34e21ab6886699f"
server_key_iv: "b3d2aa84bd8f81899d6e2c923e3d165f"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:57.944977 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:06:57.962392 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:57.964125 14114 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:06:57.964357 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "e71f8245c0194be19ad6e098599d33e6"
format_stamp: "Formatted at 2026-06-21 07:06:57 on dist-test-slave-hlq1"
server_key: "8f802827c4d38330c34e21ab6886699f"
server_key_iv: "b3d2aa84bd8f81899d6e2c923e3d165f"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:57.964620 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:57.988407 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:57.992527 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:57.992865 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:58.014009 14114 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40357
I20260621 07:06:58.014019 14180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40357 every 8 connection(s)
I20260621 07:06:58.015619 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:06:58.019577 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:06:58.022331 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14114
I20260621 07:06:58.022547 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:06:58.023375 1779 external_mini_cluster.cc:1428] Setting key a5aa020deef9a91ae9640b8142ac43b5
I20260621 07:06:58.032231 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Bootstrap starting.
I20260621 07:06:58.036070 14181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Neither blocks nor log segments found. Creating new log.
I20260621 07:06:58.037407 14181 log.cc:824] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:06:58 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:06:58.040705 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: No bootstrap required, opened a new log
I20260621 07:06:58.047350 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:06:58.025555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.1.188.254:40357)
0621 07:06:58.026358 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0621 07:06:58.026370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:06:58.026409 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:06:58.028103 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:06:58.028124 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:06:58.028524 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:06:58.028847 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:06:58.030535 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:58.031932 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:58.033786 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:06:58.034473 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:06:58.034603 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:06:58.039752 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:06:58.039761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:06:58.039777 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:06:58.039814 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:06:58.043268 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:58.044005 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:58.044011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:58.044021 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:58.044185 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:06:58.044573 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:06:58.044579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:06:58.044583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:06:58.045119 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:06:58.045445 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 07:06:58.045889 (+ 444us) server_negotiation.cc:299] Negotiation successful
0621 07:06:58.046328 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":384,"threads_started":1}
I20260621 07:06:58.048898 14181 raft_consensus.cc:348] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:06:58.049250 14181 raft_consensus.cc:374] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:06:58.049381 14181 raft_consensus.cc:729] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71f8245c0194be19ad6e098599d33e6, State: Initialized, Role: FOLLOWER
I20260621 07:06:58.050031 14181 consensus_queue.cc:260] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:06:58.050305 14181 raft_consensus.cc:388] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:06:58.050441 14181 raft_consensus.cc:482] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:06:58.050561 14181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:06:58.053017 14181 raft_consensus.cc:504] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:06:58.053448 14181 leader_election.cc:302] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e71f8245c0194be19ad6e098599d33e6; no voters:
I20260621 07:06:58.054013 14181 leader_election.cc:288] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:06:58.054199 14186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:06:58.054917 14186 raft_consensus.cc:686] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 LEADER]: Becoming Leader. State: Replica: e71f8245c0194be19ad6e098599d33e6, State: Running, Role: LEADER
I20260621 07:06:58.055519 14186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:06:58.056203 14181 sys_catalog.cc:564] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:06:58.058401 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e71f8245c0194be19ad6e098599d33e6. Latest consensus state: current_term: 1 leader_uuid: "e71f8245c0194be19ad6e098599d33e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } } }
I20260621 07:06:58.058794 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:06:58.059370 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e71f8245c0194be19ad6e098599d33e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } } }
I20260621 07:06:58.059692 14195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:06:58.059741 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:06:58.067344 14195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:06:58.068670 14181 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:06:58.086302 14195 catalog_manager.cc:1347] Generated new cluster ID: 21c3d525b0834913862ecb573484b1b6
I20260621 07:06:58.086500 14195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:06:58.106505 14195 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:06:58.108013 14195 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:06:58.122349 14195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Generated new TSK 0
I20260621 07:06:58.123536 14195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:06:58.885432 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:06:59 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:06:59.316978 14227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:06:59.317378 14227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:06:59.317584 14227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:06:59.329325 14227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:06:59.329478 14227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:06:59.329720 14227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:06:59.341822 14227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:06:59.343519 14227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:06:59.345189 14227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:06:59.363719 14237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:59.364190 14238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:06:59.376995 14240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:06:59.377795 14227 server_base.cc:1034] running on GCE node
I20260621 07:06:59.378604 14227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:06:59.379855 14227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:06:59.381108 14227 hybrid_clock.cc:648] HybridClock initialized: now 1782025619381062 us; error 48 us; skew 500 ppm
I20260621 07:06:59.381507 14227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:06:59 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:06:59.385529 14227 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:06:59.396977 14227 webserver.cc:460] Webserver started at http://127.1.188.193:45015/ using document root <none> and password file <none>
I20260621 07:06:59.397804 14227 fs_manager.cc:359] Metadata directory not provided
I20260621 07:06:59.397956 14227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:06:59.398278 14227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:06:59 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:59.427055 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "11f88caea8df4c13899170329360a750"
format_stamp: "Formatted at 2026-06-21 07:06:59 on dist-test-slave-hlq1"
server_key: "89ba288347533f2e0ed4cd65c29539f4"
server_key_iv: "f60ea457115a5db6fe02b3920891ff07"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:59.428694 14227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "11f88caea8df4c13899170329360a750"
format_stamp: "Formatted at 2026-06-21 07:06:59 on dist-test-slave-hlq1"
server_key: "89ba288347533f2e0ed4cd65c29539f4"
server_key_iv: "f60ea457115a5db6fe02b3920891ff07"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:59.439368 14227 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.001s
I20260621 07:06:59.469980 14247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:06:59.472139 14227 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.005s
I20260621 07:06:59.472442 14227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "11f88caea8df4c13899170329360a750"
format_stamp: "Formatted at 2026-06-21 07:06:59 on dist-test-slave-hlq1"
server_key: "89ba288347533f2e0ed4cd65c29539f4"
server_key_iv: "f60ea457115a5db6fe02b3920891ff07"
server_key_version: "kuduclusterkey@0"
I20260621 07:06:59.472831 14227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:06:59.541850 14227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:06:59.547852 14227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:06:59.548285 14227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:06:59.557405 14227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Jun 21 07:06:59 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:06:59.598109 14227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:06:59.598407 14227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 07:06:59.598608 14227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:06:59.598730 14227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 07:07:00 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:00.242307 14227 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:46745
I20260621 07:07:00.243474 14361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:46745 every 8 connection(s)
I20260621 07:07:00.244345 14227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:00.245736 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14227
I20260621 07:07:00.245958 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:07:00.246836 1779 external_mini_cluster.cc:1428] Setting key a39002a96d79150424fee74fe8bf13de
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Jun 21 07:07:00 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025619, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
I20260621 07:07:00.306447 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:00.259881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:58837 (local address 127.1.188.254:40357)
0621 07:07:00.266857 (+ 6976us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:00.266868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:00.270708 (+ 3840us) server_negotiation.cc:407] Connection header received
0621 07:07:00.275902 (+ 5194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:00.275909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:00.276013 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:00.276144 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:00.278879 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.280358 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.282918 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.283688 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.283811 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:00.297930 (+ 14119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:00.297939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:00.297945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:00.298011 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:00.302059 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:00.303219 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:00.303226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:00.303231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:00.303435 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:00.304069 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:00.304075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:00.304079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:00.304497 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:00.304627 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:00.305745 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0621 07:07:00.305999 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6780,"thread_start_us":6616,"threads_started":1}
I20260621 07:07:00.309058 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:00.259210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.193:58837)
0621 07:07:00.270365 (+ 11155us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:00.270428 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:00.275325 (+ 4897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:00.276649 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:00.276664 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:00.277557 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:00.278548 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:00.278579 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.280592 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.280598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:00.282713 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:00.282739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.284192 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.284199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:00.284509 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:00.292931 (+ 8422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:00.292970 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:00.297577 (+ 4607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:00.302306 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:00.302317 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:00.302340 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:00.303037 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:00.303610 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:00.303616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:00.303619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:00.303897 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:00.304754 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:00.304764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:00.305271 (+ 507us) client_negotiation.cc:769] Sending connection context
0621 07:07:00.306897 (+ 1626us) client_negotiation.cc:240] Negotiation successful
0621 07:07:00.307533 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10777,"thread_start_us":10517,"threads_started":1}
I20260621 07:07:00.311470 14362 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:00.311846 14362 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:00.312609 14362 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:00.315088 14146 ts_manager.cc:194] Registered new tserver with Master: 11f88caea8df4c13899170329360a750 (127.1.188.193:46745)
I20260621 07:07:00.317442 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:58837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:07:00.346720 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:00.719947 14368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:00.720237 14368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:00.720391 14368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:00.728075 14368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:00.728214 14368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:00.728394 14368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:00.739187 14368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:00.740653 14368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:00.741991 14368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:00.755802 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:00.755761 14383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:00.755714 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:00.757004 14368 server_base.cc:1034] running on GCE node
I20260621 07:07:00.757665 14368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:00.758775 14368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:00.760008 14368 hybrid_clock.cc:648] HybridClock initialized: now 1782025620759983 us; error 135 us; skew 500 ppm
I20260621 07:07:00.760417 14368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:00 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:00.763693 14368 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:00.765334 14368 webserver.cc:460] Webserver started at http://127.1.188.194:34063/ using document root <none> and password file <none>
I20260621 07:07:00.766027 14368 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:00.766129 14368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:00.766390 14368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:07:00 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:00.782924 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "45e9ce9e31e14e8aa3adaa9442750e8f"
format_stamp: "Formatted at 2026-06-21 07:07:00 on dist-test-slave-hlq1"
server_key: "fa3963251e3ed8166f9c9432a54e3f59"
server_key_iv: "aad2f04990cd621cb76cfaeba4634c24"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:00.784052 14368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "45e9ce9e31e14e8aa3adaa9442750e8f"
format_stamp: "Formatted at 2026-06-21 07:07:00 on dist-test-slave-hlq1"
server_key: "fa3963251e3ed8166f9c9432a54e3f59"
server_key_iv: "aad2f04990cd621cb76cfaeba4634c24"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:00.790843 14368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:07:00.806780 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:00.808380 14368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:00.808558 14368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "45e9ce9e31e14e8aa3adaa9442750e8f"
format_stamp: "Formatted at 2026-06-21 07:07:00 on dist-test-slave-hlq1"
server_key: "fa3963251e3ed8166f9c9432a54e3f59"
server_key_iv: "aad2f04990cd621cb76cfaeba4634c24"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:00.808815 14368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:00.822459 14368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:00.826610 14368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:00.826926 14368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:00.828015 14368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:00.829417 14368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:00.829488 14368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:00.829638 14368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:00.829697 14368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:00.864677 14368 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43633
I20260621 07:07:00.864725 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43633 every 8 connection(s)
I20260621 07:07:00.865978 14368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:00.868974 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14368
I20260621 07:07:00.869155 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:07:00.869800 1779 external_mini_cluster.cc:1428] Setting key d013490f3414f23c45b6be188f641573
Jun 21 07:07:00 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025620, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:00.887365 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:00.868629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:57411 (local address 127.1.188.254:40357)
0621 07:07:00.869225 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:00.869233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:00.870200 (+ 967us) server_negotiation.cc:407] Connection header received
0621 07:07:00.872568 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:00.872573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:00.872670 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:00.872783 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:00.874207 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.874951 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.876586 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.877027 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.877163 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:00.880792 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:00.880797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:00.880800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:00.880832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:00.883881 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:00.884785 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:00.884790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:00.884793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:00.884919 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:00.885420 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:00.885430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:00.885432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:00.885747 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:00.885930 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:00.886700 (+ 770us) server_negotiation.cc:299] Negotiation successful
0621 07:07:00.886956 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
I20260621 07:07:00.888502 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:00.869210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.194:57411)
0621 07:07:00.870018 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:00.870061 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:00.872309 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:00.873025 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:00.873033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:00.873466 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:00.874034 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:00.874053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.875103 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.875110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:00.876416 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:00.876435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:00.877261 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:00.877271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:00.877503 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:00.878324 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:00.878355 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:00.880563 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:00.884062 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:00.884072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:00.884092 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:00.884625 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:00.885042 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:00.885048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:00.885052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:00.885287 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:00.885889 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:00.885899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:00.886287 (+ 388us) client_negotiation.cc:769] Sending connection context
0621 07:07:00.886578 (+ 291us) client_negotiation.cc:240] Negotiation successful
0621 07:07:00.887164 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":328,"threads_started":1}
I20260621 07:07:00.890152 14503 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:00.890532 14503 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:00.891578 14503 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:07:00.893306 14146 ts_manager.cc:194] Registered new tserver with Master: 45e9ce9e31e14e8aa3adaa9442750e8f (127.1.188.194:43633)
I20260621 07:07:00.894101 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:57411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:07:00.922209 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:01.253428 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:01.253700 14509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:01.253861 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:01.261416 14509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:01.261523 14509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:01.261691 14509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:01.271611 14509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:01.272912 14509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:01.274183 14509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:01.287477 14514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:01.287499 14515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:01.287536 14517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:01.288583 14509 server_base.cc:1034] running on GCE node
I20260621 07:07:01.289216 14509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:01.290210 14509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:01.291489 14509 hybrid_clock.cc:648] HybridClock initialized: now 1782025621291399 us; error 69 us; skew 500 ppm
I20260621 07:07:01.291844 14509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:01.295136 14509 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:01.296615 14509 webserver.cc:460] Webserver started at http://127.1.188.195:42429/ using document root <none> and password file <none>
I20260621 07:07:01.297287 14509 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:01.297426 14509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:01.297734 14509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:01.312614 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "ef74f11008ad4963a3b13798d9556456"
format_stamp: "Formatted at 2026-06-21 07:07:01 on dist-test-slave-hlq1"
server_key: "9c890d1cc1ff436c76d47ff3b6b2cdbf"
server_key_iv: "28c00cbb9e37a36bf802a8720974bcde"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:01.313481 14509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "ef74f11008ad4963a3b13798d9556456"
format_stamp: "Formatted at 2026-06-21 07:07:01 on dist-test-slave-hlq1"
server_key: "9c890d1cc1ff436c76d47ff3b6b2cdbf"
server_key_iv: "28c00cbb9e37a36bf802a8720974bcde"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:01.318620 14509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:07:01.321354 14362 heartbeater.cc:503] Master 127.1.188.254:40357 was elected leader, sending a full tablet report...
I20260621 07:07:01.334833 14524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:01.336284 14509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:01.336441 14509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "ef74f11008ad4963a3b13798d9556456"
format_stamp: "Formatted at 2026-06-21 07:07:01 on dist-test-slave-hlq1"
server_key: "9c890d1cc1ff436c76d47ff3b6b2cdbf"
server_key_iv: "28c00cbb9e37a36bf802a8720974bcde"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:01.336644 14509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:01.352993 14509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:01.356993 14509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:01.357288 14509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:01.358201 14509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:01.359655 14509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:01.359725 14509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:01.359838 14509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:01.359889 14509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:01.393201 14509 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:44857
I20260621 07:07:01.393235 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:44857 every 8 connection(s)
I20260621 07:07:01.394459 14509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:01.395663 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14509
I20260621 07:07:01.395833 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:07:01.396404 1779 external_mini_cluster.cc:1428] Setting key b6a32736ebd569465cfe55d99c98e795
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025621, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:01.415572 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:01.397205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:54767 (local address 127.1.188.254:40357)
0621 07:07:01.397787 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:01.397793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:01.398730 (+ 937us) server_negotiation.cc:407] Connection header received
0621 07:07:01.400801 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:01.400808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:01.400900 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:01.401013 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:01.402387 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.403096 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.404756 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.405169 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.405300 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:01.408744 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:01.408749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:01.408753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:01.408790 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:01.412407 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.413195 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.413199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.413201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.413368 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.413829 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.413834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.413838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.414113 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:01.414265 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:01.414959 (+ 694us) server_negotiation.cc:299] Negotiation successful
0621 07:07:01.415180 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
I20260621 07:07:01.416563 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:01.397727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.195:54767)
0621 07:07:01.398572 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:01.398619 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:01.400546 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:01.401258 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:01.401266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:01.401670 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:01.402228 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:01.402245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.403249 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.403254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:01.404602 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:01.404625 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.405380 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.405387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:01.405640 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:01.406333 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:01.406355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:01.408506 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:01.412543 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:01.412553 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:01.412576 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:01.413008 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:01.413474 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:01.413479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:01.413483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:01.413706 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:01.414241 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:01.414249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:01.414590 (+ 341us) client_negotiation.cc:769] Sending connection context
0621 07:07:01.414862 (+ 272us) client_negotiation.cc:240] Negotiation successful
0621 07:07:01.415422 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":328,"threads_started":1}
I20260621 07:07:01.417953 14637 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:01.418260 14637 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:01.418918 14637 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:01.420337 14146 ts_manager.cc:194] Registered new tserver with Master: ef74f11008ad4963a3b13798d9556456 (127.1.188.195:44857)
I20260621 07:07:01.421103 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:54767
I20260621 07:07:01.424770 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:07:01.426765 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14227
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:07:01.444955 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:01.427758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.1.188.193:46745)
0621 07:07:01.428454 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:01.428464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:01.428499 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:07:01.428586 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:01.428748 (+ 162us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:01.428977 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:01.429266 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:01.430843 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.432030 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.433518 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.434090 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.434241 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:01.438570 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:01.438584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:01.438602 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:01.438642 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:01.441931 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.442719 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.442733 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.442735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.442870 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.443263 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.443268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.443273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.443686 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:01.444010 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:01.444335 (+ 325us) server_negotiation.cc:299] Negotiation successful
0621 07:07:01.444605 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":282,"threads_started":1}
I20260621 07:07:01.446285 14357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:01.529954 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14227
I20260621 07:07:01.545660 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14368
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:07:01.562853 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:01.546545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.1.188.194:43633)
0621 07:07:01.547075 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:01.547082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:01.547105 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:07:01.547172 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:01.547179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:01.547387 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:01.547600 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:01.548913 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.549945 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.551726 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.552271 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.552412 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:01.556841 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:01.556854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:01.556868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:01.556912 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:01.559982 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.560723 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.560730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.560733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.560863 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.561245 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.561251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.561254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.561588 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:01.561881 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:01.562207 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 07:07:01.562497 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
I20260621 07:07:01.564226 14498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:01.651885 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14368
I20260621 07:07:01.672096 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14509
Jun 21 07:07:01 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:07:01.688973 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:01.673058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.1.188.195:44857)
0621 07:07:01.673728 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:01.673735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:01.673765 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:07:01.673825 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:01.673833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:01.674053 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:01.674230 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:01.675724 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.676736 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.678042 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:01.678599 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:01.678745 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:01.683144 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:01.683155 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:01.683170 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:01.683207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:01.686020 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.686730 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.686738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.686740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.686874 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:01.687248 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:01.687253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:01.687256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:01.687657 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:01.687987 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:01.688306 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 07:07:01.688582 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":373,"threads_started":1}
I20260621 07:07:01.690316 14632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:01.774032 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14509
I20260621 07:07:01.789664 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14114
I20260621 07:07:01.790982 14176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:01.897089 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14114
I20260621 07:07:01.920943 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40357
--webserver_interface=127.1.188.254
--webserver_port=37703
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40357
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:02.243577 14649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:02.243913 14649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:02.244081 14649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:02.251446 14649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:02.251585 14649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:02.251693 14649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:02.251796 14649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:02.251886 14649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:02.261852 14649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40357
--ranger_config_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=37703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:02.263178 14649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:02.264528 14649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:02.276026 14657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:02.276151 14654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:02.276085 14655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:02.277037 14649 server_base.cc:1034] running on GCE node
I20260621 07:07:02.277854 14649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:02.279359 14649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:02.280632 14649 hybrid_clock.cc:648] HybridClock initialized: now 1782025622280573 us; error 39 us; skew 500 ppm
I20260621 07:07:02.280990 14649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:02 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:02.284128 14649 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:02.285485 14649 webserver.cc:460] Webserver started at http://127.1.188.254:37703/ using document root <none> and password file <none>
I20260621 07:07:02.286070 14649 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:02.286175 14649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:07:02 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:02.336829 14649 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:07:02.339993 14665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:02.341432 14649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:07:02.341630 14649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "e71f8245c0194be19ad6e098599d33e6"
format_stamp: "Formatted at 2026-06-21 07:06:57 on dist-test-slave-hlq1"
server_key: "8f802827c4d38330c34e21ab6886699f"
server_key_iv: "b3d2aa84bd8f81899d6e2c923e3d165f"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:02.342559 14649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:02.371620 14649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:02.375636 14649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:02.375931 14649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:02.394788 14649 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40357
I20260621 07:07:02.394812 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40357 every 8 connection(s)
I20260621 07:07:02.396324 14649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:02.403026 14717 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:07:02.403956 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14649
I20260621 07:07:02.404794 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:46745
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:40357
--webserver_port=45015
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:07:02.406541 14717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Bootstrap starting.
I20260621 07:07:02.425621 14717 log.cc:824] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:02.434664 14717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:07:02.435199 14717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Bootstrap complete.
I20260621 07:07:02.444782 14717 raft_consensus.cc:348] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:07:02.445719 14717 raft_consensus.cc:729] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71f8245c0194be19ad6e098599d33e6, State: Initialized, Role: FOLLOWER
I20260621 07:07:02.446346 14717 consensus_queue.cc:260] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:07:02.446595 14717 raft_consensus.cc:388] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:02.446725 14717 raft_consensus.cc:482] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:02.446923 14717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:07:02.450697 14717 raft_consensus.cc:504] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:07:02.451268 14717 leader_election.cc:302] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e71f8245c0194be19ad6e098599d33e6; no voters:
I20260621 07:07:02.451933 14717 leader_election.cc:288] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:07:02.452068 14721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:07:02.454321 14717 sys_catalog.cc:564] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:02.455690 14721 raft_consensus.cc:686] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [term 2 LEADER]: Becoming Leader. State: Replica: e71f8245c0194be19ad6e098599d33e6, State: Running, Role: LEADER
I20260621 07:07:02.456414 14721 consensus_queue.cc:237] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } }
I20260621 07:07:02.459331 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e71f8245c0194be19ad6e098599d33e6. Latest consensus state: current_term: 2 leader_uuid: "e71f8245c0194be19ad6e098599d33e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } } }
I20260621 07:07:02.459728 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:02.460335 14722 sys_catalog.cc:455] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e71f8245c0194be19ad6e098599d33e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e71f8245c0194be19ad6e098599d33e6" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40357 } } }
I20260621 07:07:02.460604 14722 sys_catalog.cc:458] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:02.465180 14731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:02.471051 14731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:02.471879 14717 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 07:07:02.472847 14731 catalog_manager.cc:1259] Loaded cluster ID: 21c3d525b0834913862ecb573484b1b6
I20260621 07:07:02.472977 14731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:02.478168 14731 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:02.480024 14731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Loaded TSK: 0
I20260621 07:07:02.487185 14731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:07:02.859069 14719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:02.859499 14719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:02.859681 14719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:02.871827 14719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:02.872040 14719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:02.872258 14719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:07:02.888441 14719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:46745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=45015
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:02.890385 14719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:02.892220 14719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:02.919849 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:02.927932 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:02.940976 14719 server_base.cc:1034] running on GCE node
W20260621 07:07:02.941541 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:02.942554 14719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:02.959765 14719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:02.961248 14719 hybrid_clock.cc:648] HybridClock initialized: now 1782025622961162 us; error 92 us; skew 500 ppm
I20260621 07:07:02.961715 14719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:02 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:02.966249 14719 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:07:02.989831 14719 webserver.cc:460] Webserver started at http://127.1.188.193:45015/ using document root <none> and password file <none>
I20260621 07:07:02.991041 14719 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:02.991326 14719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:07:03 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:03.054780 14719 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.003s sys 0.016s
I20260621 07:07:03.072861 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:03.074916 14719 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.014s
I20260621 07:07:03.075183 14719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "11f88caea8df4c13899170329360a750"
format_stamp: "Formatted at 2026-06-21 07:06:59 on dist-test-slave-hlq1"
server_key: "89ba288347533f2e0ed4cd65c29539f4"
server_key_iv: "f60ea457115a5db6fe02b3920891ff07"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:03.076785 14719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:03.116075 14719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:03.121649 14719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:03.122053 14719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:03.123366 14719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:03.138267 14719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:03.138502 14719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 07:07:03.138725 14719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:03.138880 14719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:03.252071 14719 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:46745
I20260621 07:07:03.252158 14878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:46745 every 8 connection(s)
I20260621 07:07:03.254294 14719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:03.256446 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14719
I20260621 07:07:03.257332 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:43633
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:40357
--webserver_port=34063
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:07:03 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025622, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:03.385448 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:03.269034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:35401 (local address 127.1.188.254:40357)
0621 07:07:03.312790 (+ 43756us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:03.312822 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:03.336387 (+ 23565us) server_negotiation.cc:407] Connection header received
0621 07:07:03.343129 (+ 6742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:03.343220 (+ 91us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:03.343942 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:03.344413 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:03.348861 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:03.350854 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:03.357116 (+ 6262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:03.358273 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:03.358446 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:03.374166 (+ 15720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:03.374182 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:03.374208 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:03.374275 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:03.378131 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:03.379189 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:03.379199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:03.379207 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:03.379427 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:03.379968 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:03.379973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:03.379976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:03.380601 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:03.380786 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:03.383746 (+ 2960us) server_negotiation.cc:299] Negotiation successful
0621 07:07:03.384174 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43313,"thread_start_us":43654,"threads_started":1}
I20260621 07:07:03.383369 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:03.267811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.193:35401)
0621 07:07:03.336129 (+ 68318us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:03.336201 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:03.342217 (+ 6016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:03.345472 (+ 3255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:03.345486 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:03.347805 (+ 2319us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:03.348611 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:03.348637 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:03.352338 (+ 3701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:03.352343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:03.354442 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:03.354462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:03.360732 (+ 6270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:03.360740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:03.361050 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:03.367683 (+ 6633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:03.367719 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:03.373758 (+ 6039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:03.378350 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:03.378359 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:03.378373 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:03.378978 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:03.379564 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:03.379569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:03.379572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:03.379829 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:03.380871 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:03.380878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:03.381279 (+ 401us) client_negotiation.cc:769] Sending connection context
0621 07:07:03.381567 (+ 288us) client_negotiation.cc:240] Negotiation successful
0621 07:07:03.382117 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67975,"thread_start_us":67719,"threads_started":1}
I20260621 07:07:03.390651 14879 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:03.390985 14879 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:03.391868 14879 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
W20260621 07:07:03.396116 14879 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:40357 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:07:03.396409 14879 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:03.396795 14879 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:03.408488 14879 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:03.408885 14879 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
W20260621 07:07:03.410724 14879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:07:03.431596 14890 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260621 07:07:03.785441 14881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:03.785840 14881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:03.786024 14881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:03.797762 14881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:03.797914 14881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:03.798163 14881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:03.813868 14881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:43633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=34063
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:03.815599 14881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:03.817296 14881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:03.834029 14899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:03.834789 14900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Jun 21 07:07:03 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 07:07:03.839758 14902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:03.841106 14881 server_base.cc:1034] running on GCE node
I20260621 07:07:03.841958 14881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:03.843331 14881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:03.844646 14881 hybrid_clock.cc:648] HybridClock initialized: now 1782025623844587 us; error 63 us; skew 500 ppm
I20260621 07:07:03.845058 14881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:03 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:03.849521 14881 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:03.864982 14881 webserver.cc:460] Webserver started at http://127.1.188.194:34063/ using document root <none> and password file <none>
I20260621 07:07:03.866029 14881 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:03.866271 14881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:07:03 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:03.895054 14881 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:07:03.898133 14910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:03.900071 14881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260621 07:07:03.900259 14881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "45e9ce9e31e14e8aa3adaa9442750e8f"
format_stamp: "Formatted at 2026-06-21 07:07:00 on dist-test-slave-hlq1"
server_key: "fa3963251e3ed8166f9c9432a54e3f59"
server_key_iv: "aad2f04990cd621cb76cfaeba4634c24"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:03.901495 14881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:03.936075 14881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:03.941733 14881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:03.942196 14881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:03.943526 14881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:03.945370 14881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:03.945489 14881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:03.945613 14881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:03.945693 14881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:04.001303 14881 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:43633
I20260621 07:07:04.001362 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:43633 every 8 connection(s)
I20260621 07:07:04.003377 14881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:04.013507 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 14881
I20260621 07:07:04.014482 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.188.212:44075/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:44857
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:40357
--webserver_port=42429
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:04.064314 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.012598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:35303 (local address 127.1.188.254:40357)
0621 07:07:04.034547 (+ 21949us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.034555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.038471 (+ 3916us) server_negotiation.cc:407] Connection header received
0621 07:07:04.038550 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.038556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.038656 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.038767 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:04.046745 (+ 7978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.047947 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.050535 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.051205 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.051316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.056313 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:04.056322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:04.056328 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:04.056380 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:04.060237 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.061282 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.061288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.061292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.061455 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.062056 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.062061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.062065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.062440 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:04.062562 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.063678 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.063919 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21775,"thread_start_us":30810,"threads_started":1}
I20260621 07:07:04.066942 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.011793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.194:35303)
0621 07:07:04.035198 (+ 23405us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:04.035256 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:04.038006 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:04.044771 (+ 6765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:04.044786 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:04.045492 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:04.046333 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.046360 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.048187 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.048194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.050337 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.050355 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.051774 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.051785 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.052098 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.052974 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:04.053007 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:04.055966 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:04.060420 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:04.060430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:04.060449 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:04.061121 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:04.061629 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:04.061633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:04.061636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:04.061914 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:04.062698 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:04.062707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:04.063174 (+ 467us) client_negotiation.cc:769] Sending connection context
0621 07:07:04.064779 (+ 1605us) client_negotiation.cc:240] Negotiation successful
0621 07:07:04.065516 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23019,"thread_start_us":23060,"threads_started":1}
I20260621 07:07:04.068861 15024 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:04.069206 15024 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:04.070040 15024 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:04.072188 14682 ts_manager.cc:194] Registered new tserver with Master: 45e9ce9e31e14e8aa3adaa9442750e8f (127.1.188.194:43633)
I20260621 07:07:04.074505 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:35303
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:04.410836 14879 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:04.411348 14879 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:04.412755 14682 ts_manager.cc:194] Registered new tserver with Master: 11f88caea8df4c13899170329360a750 (127.1.188.193:46745)
I20260621 07:07:04.414032 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:35401
W20260621 07:07:04.510430 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:04.510933 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:04.511168 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:04.523195 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:04.523447 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:04.523686 15028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:04.534678 15028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.188.212:44075/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:44857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=42429
--tserver_master_addrs=127.1.188.254:40357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:04.536345 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:04.538177 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:04.558372 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:04.559415 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:04.564471 15028 server_base.cc:1034] running on GCE node
W20260621 07:07:04.564785 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:04.565644 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:04.566911 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:04.568233 15028 hybrid_clock.cc:648] HybridClock initialized: now 1782025624568166 us; error 60 us; skew 500 ppm
I20260621 07:07:04.568629 15028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:04.573253 15028 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:04.589563 15028 webserver.cc:460] Webserver started at http://127.1.188.195:42429/ using document root <none> and password file <none>
I20260621 07:07:04.590783 15028 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:04.590981 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025623, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for HTTP/127.1.188.212@KRBTEST.COM
I20260621 07:07:04.632771 15028 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260621 07:07:04.644074 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:04.646055 15028 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.008s
I20260621 07:07:04.646306 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "ef74f11008ad4963a3b13798d9556456"
format_stamp: "Formatted at 2026-06-21 07:07:01 on dist-test-slave-hlq1"
server_key: "9c890d1cc1ff436c76d47ff3b6b2cdbf"
server_key_iv: "28c00cbb9e37a36bf802a8720974bcde"
server_key_version: "kuduclusterkey@0"
I20260621 07:07:04.647545 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:04.689442 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:04.694931 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:04.695323 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:04.703065 15028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:04.711516 15028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:04.711712 15028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:04.711930 15028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:04.712062 15028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:04.828749 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:44857 every 8 connection(s)
I20260621 07:07:04.829030 15028 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:44857
I20260621 07:07:04.831182 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:04.841914 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15028
Jun 21 07:07:04 dist-test-slave-hlq1 krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025624, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:04.896189 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.840174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:46903 (local address 127.1.188.254:40357)
0621 07:07:04.874668 (+ 34494us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.874674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.874694 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:07:04.874772 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.874780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.874910 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.875061 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:04.877191 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.878737 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.882060 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.882799 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.882950 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.888413 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:04.888420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:04.888425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:04.888470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:04.892096 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.893032 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.893037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.893041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.893181 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.893692 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.893697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.893701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.894048 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:04.894198 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.895592 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.895813 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34374,"thread_start_us":34244,"threads_started":1}
I20260621 07:07:04.898164 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.840866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:40357 (local address 127.1.188.195:46903)
0621 07:07:04.868908 (+ 28042us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:04.868988 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:04.873857 (+ 4869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:04.875630 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:04.875640 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:04.876109 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:04.876993 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.877014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.878986 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.878994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.881804 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.881827 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.883338 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.883347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.883706 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.884554 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:04.884587 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:04.888136 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:04.892265 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:04.892275 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:04.892293 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:04.892886 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:04.893311 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:04.893316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:04.893320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:04.893545 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:04.894278 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:04.894286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:04.894711 (+ 425us) client_negotiation.cc:769] Sending connection context
0621 07:07:04.895007 (+ 296us) client_negotiation.cc:240] Negotiation successful
0621 07:07:04.896806 (+ 1799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27609,"thread_start_us":30621,"threads_started":1}
I20260621 07:07:04.900192 15158 heartbeater.cc:347] Connected to a master server at 127.1.188.254:40357
I20260621 07:07:04.900491 15158 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:04.901283 15158 heartbeater.cc:511] Master 127.1.188.254:40357 requested a full tablet report, sending...
I20260621 07:07:04.901572 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.874646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.1.188.254:40357)
0621 07:07:04.883522 (+ 8876us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.883528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.883546 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:07:04.883599 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.883604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.883687 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.883811 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:04.886179 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.887455 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.889469 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.890195 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.893723 (+ 3528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.894013 (+ 290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:04.894019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:04.894024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:04.894061 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:04.897521 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.898389 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.898395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.898398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.898541 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.899024 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.899029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.899033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.899422 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:04.900863 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.900996 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.901211 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8732,"thread_start_us":9341,"threads_started":1}
I20260621 07:07:04.903306 14682 ts_manager.cc:194] Registered new tserver with Master: ef74f11008ad4963a3b13798d9556456 (127.1.188.195:44857)
I20260621 07:07:04.904507 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:46903
I20260621 07:07:04.905642 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:07:04.924577 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.910598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.1.188.254:40357)
0621 07:07:04.910781 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.910785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.910804 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:07:04.910975 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.910984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.911060 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.911129 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:04.913087 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.913785 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.915721 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.916192 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.916274 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.918957 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:04.918963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:04.918968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:04.919010 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:04.921749 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.922506 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.922512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.922516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.922664 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:04.923117 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:04.923123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:04.923127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:04.923497 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:04.923758 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.924005 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.924234 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:07:04.929759 14681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:07:04.937456 14681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:07:04.962877 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.953017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44857 (local address 127.0.0.1:59336)
0621 07:07:04.953788 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:04.953831 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:04.953998 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:04.955228 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:04.955235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:04.955278 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:04.955729 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.955741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.957912 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.957918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.960153 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.960168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.961922 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.961927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.962094 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.962151 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 07:07:04.962254 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 07:07:04.962413 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":557,"threads_started":1}
I20260621 07:07:04.964743 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.953187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.1.188.195:44857)
0621 07:07:04.954599 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.954607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.954635 (+ 28us) server_negotiation.cc:407] Connection header received
0621 07:07:04.954732 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.954739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.954959 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.955925 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:04.956195 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.957759 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.960345 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.961706 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.963777 (+ 2071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.964064 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.964145 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.964344 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1100,"threads_started":1}
I20260621 07:07:04.969612 15093 tablet_service.cc:1467] Processing CreateTablet for tablet 49738085b39a4f52978e71a52a8589eb (DEFAULT_TABLE table=test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:04.972295 15093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49738085b39a4f52978e71a52a8589eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:04.973709 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.954552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:46745 (local address 127.0.0.1:59378)
0621 07:07:04.955032 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:04.955049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:04.955163 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:04.966400 (+ 11237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:04.966406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:04.966432 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:04.966791 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.966807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.968770 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.968776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.970803 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.970823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.972922 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.972928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.973090 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.973122 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:07:04.973214 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:07:04.973343 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":646,"threads_started":1}
I20260621 07:07:04.974385 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.955039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.1.188.193:46745)
0621 07:07:04.965806 (+ 10767us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.965819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.965857 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:07:04.965964 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.965973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.966216 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.966412 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:04.967008 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.968587 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.970993 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.972428 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.972557 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.972782 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.973920 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.974091 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10582,"threads_started":1}
I20260621 07:07:04.974021 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.955595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:43633 (local address 127.0.0.1:45416)
0621 07:07:04.963147 (+ 7552us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:04.963167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:04.963286 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:04.967483 (+ 4197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:04.967487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:04.967513 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:04.967672 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.967681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.969640 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.969645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.971586 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:04.971600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.973308 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.973314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:04.973521 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.973556 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 07:07:04.973615 (+ 59us) client_negotiation.cc:240] Negotiation successful
0621 07:07:04.973758 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7438,"thread_start_us":9638,"threads_started":1}
I20260621 07:07:04.974851 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:04.955764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.1.188.194:43633)
0621 07:07:04.966945 (+ 11181us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:04.966954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:04.966978 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:07:04.967063 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:04.967070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:04.967265 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:04.967829 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:04.968029 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.969492 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.971790 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:04.973141 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:04.974135 (+ 994us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:04.974310 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:04.974370 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 07:07:04.974547 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11025,"threads_started":1}
I20260621 07:07:04.978498 14814 tablet_service.cc:1467] Processing CreateTablet for tablet 49738085b39a4f52978e71a52a8589eb (DEFAULT_TABLE table=test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:04.978498 14959 tablet_service.cc:1467] Processing CreateTablet for tablet 49738085b39a4f52978e71a52a8589eb (DEFAULT_TABLE table=test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:04.979943 14814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49738085b39a4f52978e71a52a8589eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:04.979959 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49738085b39a4f52978e71a52a8589eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:05.000680 15188 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Bootstrap starting.
I20260621 07:07:05.005160 15187 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Bootstrap starting.
I20260621 07:07:05.008467 15188 tablet_bootstrap.cc:654] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:05.008729 15187 tablet_bootstrap.cc:654] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:04.998459 15186 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Bootstrap starting.
I20260621 07:07:05.009874 15188 log.cc:824] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:05.011880 15187 log.cc:824] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:05.013074 15186 tablet_bootstrap.cc:654] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:05.014238 15186 log.cc:824] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:05.032711 15187 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: No bootstrap required, opened a new log
I20260621 07:07:05.033247 15187 ts_tablet_manager.cc:1397] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260621 07:07:05.042419 15188 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: No bootstrap required, opened a new log
I20260621 07:07:05.042871 15188 ts_tablet_manager.cc:1397] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.021s
I20260621 07:07:05.042950 15186 tablet_bootstrap.cc:492] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: No bootstrap required, opened a new log
I20260621 07:07:05.043236 15186 ts_tablet_manager.cc:1397] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.004s
I20260621 07:07:05.046618 15187 raft_consensus.cc:348] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.047184 15187 raft_consensus.cc:374] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:05.047335 15187 raft_consensus.cc:729] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e9ce9e31e14e8aa3adaa9442750e8f, State: Initialized, Role: FOLLOWER
I20260621 07:07:05.048154 15187 consensus_queue.cc:260] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.054391 15188 raft_consensus.cc:348] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.054937 15188 raft_consensus.cc:374] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:05.055068 15188 raft_consensus.cc:729] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f88caea8df4c13899170329360a750, State: Initialized, Role: FOLLOWER
I20260621 07:07:05.055882 15188 consensus_queue.cc:260] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.055784 15186 raft_consensus.cc:348] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.056200 15186 raft_consensus.cc:374] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:05.056325 15186 raft_consensus.cc:729] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef74f11008ad4963a3b13798d9556456, State: Initialized, Role: FOLLOWER
I20260621 07:07:05.056396 15187 ts_tablet_manager.cc:1428] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20260621 07:07:05.056828 15024 heartbeater.cc:503] Master 127.1.188.254:40357 was elected leader, sending a full tablet report...
I20260621 07:07:05.057034 15186 consensus_queue.cc:260] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.067201 14879 heartbeater.cc:503] Master 127.1.188.254:40357 was elected leader, sending a full tablet report...
I20260621 07:07:05.068158 15188 ts_tablet_manager.cc:1428] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Time spent starting tablet: real 0.025s user 0.019s sys 0.005s
I20260621 07:07:05.069360 15192 raft_consensus.cc:482] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:07:05.069697 15192 raft_consensus.cc:504] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.074018 15192 leader_election.cc:288] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef74f11008ad4963a3b13798d9556456 (127.1.188.195:44857), 11f88caea8df4c13899170329360a750 (127.1.188.193:46745)
I20260621 07:07:05.083135 15158 heartbeater.cc:503] Master 127.1.188.254:40357 was elected leader, sending a full tablet report...
I20260621 07:07:05.083877 15186 ts_tablet_manager.cc:1428] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Time spent starting tablet: real 0.040s user 0.014s sys 0.023s
I20260621 07:07:05.087847 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.072837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:34275 (local address 127.1.188.195:44857)
0621 07:07:05.073062 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.073069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.074838 (+ 1769us) server_negotiation.cc:407] Connection header received
0621 07:07:05.075175 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.075180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.075260 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.075336 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:05.076659 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.078505 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.085393 (+ 6888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.086614 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.086744 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.086826 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.087463 (+ 637us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.087591 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:07:05.088858 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.072295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:44857 (local address 127.1.188.194:34275)
0621 07:07:05.074736 (+ 2441us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:05.074758 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:05.074962 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:05.075599 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:05.075604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:05.075642 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:05.076393 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:05.076428 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.078751 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.078760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:05.081887 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:05.081910 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.087041 (+ 5131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.087048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:05.087255 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.087288 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:07:05.088261 (+ 973us) client_negotiation.cc:240] Negotiation successful
0621 07:07:05.088418 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311,"thread_start_us":2173,"threads_started":1}
I20260621 07:07:05.089989 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49738085b39a4f52978e71a52a8589eb" candidate_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef74f11008ad4963a3b13798d9556456" is_pre_election: true
I20260621 07:07:05.090552 15113 raft_consensus.cc:2454] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e9ce9e31e14e8aa3adaa9442750e8f in term 0.
I20260621 07:07:05.091254 14912 leader_election.cc:302] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9ce9e31e14e8aa3adaa9442750e8f, ef74f11008ad4963a3b13798d9556456; no voters:
I20260621 07:07:05.091681 15192 raft_consensus.cc:2781] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:07:05.091825 15192 raft_consensus.cc:482] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:07:05.091943 15192 raft_consensus.cc:3037] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:05.093087 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.075238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:50511 (local address 127.1.188.193:46745)
0621 07:07:05.075460 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.075466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.085587 (+ 10121us) server_negotiation.cc:407] Connection header received
0621 07:07:05.086026 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.086031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.086126 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.086223 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:05.086942 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.087986 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.090959 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.092148 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.092310 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.092378 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.092742 (+ 364us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.092866 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:07:05.093350 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.074899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:46745 (local address 127.1.188.194:50511)
0621 07:07:05.085485 (+ 10586us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:05.085505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:05.085862 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:05.086390 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:05.086395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:05.086423 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:05.086763 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:05.086774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.088191 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.088197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:05.090792 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:05.090810 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.092367 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.092374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:05.092559 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.092609 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 07:07:05.092706 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 07:07:05.092840 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10455,"thread_start_us":10728,"threads_started":1}
I20260621 07:07:05.094626 15192 raft_consensus.cc:504] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.095185 15192 leader_election.cc:288] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [CANDIDATE]: Term 1 election: Requested vote from peers ef74f11008ad4963a3b13798d9556456 (127.1.188.195:44857), 11f88caea8df4c13899170329360a750 (127.1.188.193:46745)
I20260621 07:07:05.095881 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49738085b39a4f52978e71a52a8589eb" candidate_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef74f11008ad4963a3b13798d9556456"
I20260621 07:07:05.096117 15113 raft_consensus.cc:3037] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:05.096484 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49738085b39a4f52978e71a52a8589eb" candidate_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f88caea8df4c13899170329360a750"
I20260621 07:07:05.096526 14834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49738085b39a4f52978e71a52a8589eb" candidate_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f88caea8df4c13899170329360a750" is_pre_election: true
I20260621 07:07:05.097070 14833 raft_consensus.cc:3037] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:05.098929 15113 raft_consensus.cc:2454] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9ce9e31e14e8aa3adaa9442750e8f in term 1.
I20260621 07:07:05.099310 14912 leader_election.cc:302] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e9ce9e31e14e8aa3adaa9442750e8f, ef74f11008ad4963a3b13798d9556456; no voters:
I20260621 07:07:05.099622 15192 raft_consensus.cc:2781] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:05.099789 15192 raft_consensus.cc:686] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 1 LEADER]: Becoming Leader. State: Replica: 45e9ce9e31e14e8aa3adaa9442750e8f, State: Running, Role: LEADER
I20260621 07:07:05.100338 15192 consensus_queue.cc:237] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } }
I20260621 07:07:05.100492 14833 raft_consensus.cc:2454] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e9ce9e31e14e8aa3adaa9442750e8f in term 1.
I20260621 07:07:05.105553 14681 catalog_manager.cc:5495] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 45e9ce9e31e14e8aa3adaa9442750e8f (127.1.188.194). New cstate: current_term: 1 leader_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e9ce9e31e14e8aa3adaa9442750e8f" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 43633 } health_report { overall_health: HEALTHY } } }
W20260621 07:07:05.117790 15162 tablet.cc:2369] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:07:05.133946 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.127747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.1.188.194:43633)
0621 07:07:05.127923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.127927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.127945 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:07:05.128088 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.128126 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.128224 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.128353 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:07:05.128791 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.129562 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.131545 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.131954 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.132064 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.132656 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:07:05.133282 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:07:05.133436 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.133558 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.133640 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:07:05.142279 14833 raft_consensus.cc:1261] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 1 FOLLOWER]: Refusing update from remote peer 45e9ce9e31e14e8aa3adaa9442750e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:05.142310 15113 raft_consensus.cc:1261] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 1 FOLLOWER]: Refusing update from remote peer 45e9ce9e31e14e8aa3adaa9442750e8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:05.143286 15195 consensus_queue.cc:1035] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f88caea8df4c13899170329360a750" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 46745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:05.143723 15192 consensus_queue.cc:1035] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef74f11008ad4963a3b13798d9556456" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:05.175272 14681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:05.175567 14681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:05.179116 14681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e71f8245c0194be19ad6e098599d33e6: Sending DeleteTablet for 3 replicas of tablet 49738085b39a4f52978e71a52a8589eb
I20260621 07:07:05.180433 15093 tablet_service.cc:1514] Processing DeleteTablet for tablet 49738085b39a4f52978e71a52a8589eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:05 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59336
I20260621 07:07:05.180863 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet 49738085b39a4f52978e71a52a8589eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:05 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:45416
I20260621 07:07:05.180871 14814 tablet_service.cc:1514] Processing DeleteTablet for tablet 49738085b39a4f52978e71a52a8589eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:05 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59378
I20260621 07:07:05.183082 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14719
I20260621 07:07:05.194947 15212 tablet_replica.cc:330] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: stopping tablet replica
I20260621 07:07:05.195773 15212 raft_consensus.cc:2229] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:05.195909 15210 tablet_replica.cc:330] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: stopping tablet replica
I20260621 07:07:05.196316 15212 raft_consensus.cc:2258] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:05.196367 15210 raft_consensus.cc:2229] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:05.196910 15210 raft_consensus.cc:2258] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:05.198451 15212 ts_tablet_manager.cc:1905] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:05.198840 15210 ts_tablet_manager.cc:1905] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:05.206399 15212 ts_tablet_manager.cc:1918] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:05.206708 15212 log.cc:1196] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/49738085b39a4f52978e71a52a8589eb
I20260621 07:07:05.207468 15212 ts_tablet_manager.cc:1939] T 49738085b39a4f52978e71a52a8589eb P 11f88caea8df4c13899170329360a750: Deleting consensus metadata
I20260621 07:07:05.208011 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.187422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.1.188.193:46745)
0621 07:07:05.187603 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.187610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.187629 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:07:05.187698 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.187703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.187774 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.187884 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:05.190054 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.191198 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.194203 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.195160 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.195333 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.198890 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:05.198903 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:05.198917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:05.198955 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:05.202823 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.203842 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.203856 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.203860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.204022 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.205127 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.205132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.205136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.205573 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:05.205891 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.207169 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.207448 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:07:05.208421 15211 tablet_replica.cc:330] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: stopping tablet replica
I20260621 07:07:05.209007 14874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:05.208986 15211 raft_consensus.cc:2229] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:07:05.209486 14666 catalog_manager.cc:4841] TS 11f88caea8df4c13899170329360a750 (127.1.188.193:46745): tablet 49738085b39a4f52978e71a52a8589eb (table test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]) successfully deleted
I20260621 07:07:05.209715 15211 raft_consensus.cc:2258] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:05.211252 15211 ts_tablet_manager.cc:1905] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:05.215337 15210 ts_tablet_manager.cc:1918] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:05.215639 15210 log.cc:1196] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/49738085b39a4f52978e71a52a8589eb
I20260621 07:07:05.216205 15211 ts_tablet_manager.cc:1918] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:05.216329 15210 ts_tablet_manager.cc:1939] T 49738085b39a4f52978e71a52a8589eb P ef74f11008ad4963a3b13798d9556456: Deleting consensus metadata
I20260621 07:07:05.216394 15211 log.cc:1196] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/49738085b39a4f52978e71a52a8589eb
I20260621 07:07:05.216877 15211 ts_tablet_manager.cc:1939] T 49738085b39a4f52978e71a52a8589eb P 45e9ce9e31e14e8aa3adaa9442750e8f: Deleting consensus metadata
I20260621 07:07:05.218422 14666 catalog_manager.cc:4841] TS 45e9ce9e31e14e8aa3adaa9442750e8f (127.1.188.194:43633): tablet 49738085b39a4f52978e71a52a8589eb (table test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]) successfully deleted
I20260621 07:07:05.218703 14666 catalog_manager.cc:4841] TS ef74f11008ad4963a3b13798d9556456 (127.1.188.195:44857): tablet 49738085b39a4f52978e71a52a8589eb (table test-table [id=167cef8ff1da4122a2bf5c47c9e431b9]) successfully deleted
I20260621 07:07:05.357427 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14719
I20260621 07:07:05.381398 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14881
I20260621 07:07:05.403973 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.382304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.1.188.194:43633)
0621 07:07:05.382472 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.382477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.382500 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:07:05.382579 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.382583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.382649 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.382730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:05.384449 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.385328 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.387058 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.387579 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.387679 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.394732 (+ 7053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:05.394751 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:05.394773 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:05.394847 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:05.398893 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.399899 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.399909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.399913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.400987 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.401451 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.401460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.401466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.401939 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:05.402247 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.403344 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.403600 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 07:07:05.407868 15019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:05.521965 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14881
I20260621 07:07:05.536787 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15028
I20260621 07:07:05.551111 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:05.537554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.1.188.195:44857)
0621 07:07:05.537739 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:05.537744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:05.537763 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:07:05.537827 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:05.537831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:05.537896 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:05.537986 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:05.539274 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.540194 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.541558 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:05.541990 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:05.542101 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:05.544728 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:05.544743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:05.544759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:05.544801 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:05.548110 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.548996 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.549003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.549005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.549190 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:05.549543 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:05.549549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:05.549552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:05.549908 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:05.550237 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:05.550533 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:07:05.550766 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:07:05.551784 15153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:05.648653 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15028
I20260621 07:07:05.667743 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 14649
I20260621 07:07:05.668957 14712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:05.780632 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 14649
I20260621 07:07:05.801900 1779 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 07:07:06.138041 1779 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 07:07:06.138319 1779 mini_ranger.cc:66] Stopping Ranger...
I20260621 07:07:06.484603 1779 mini_ranger.cc:68] Stopped Ranger
2026-06-21 07:07:06.484 UTC [13700] LOG: received smart shutdown request
2026-06-21 07:07:06.487 UTC [13700] LOG: background worker "logical replication launcher" (PID 13709) exited with exit code 1
2026-06-21 07:07:06.488 UTC [13704] LOG: shutting down
2026-06-21 07:07:06.625 UTC [13700] LOG: database system is shut down
2026-06-21T07:07:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54420 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:07:06 dist-test-slave-hlq1 krb5kdc[15225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:07:06 dist-test-slave-hlq1 krb5kdc[15225](info): set up 2 sockets
Jun 21 07:07:06 dist-test-slave-hlq1 krb5kdc[15225](info): commencing operation
krb5kdc: starting...
W20260621 07:07:07.884449 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:07:08 dist-test-slave-hlq1 krb5kdc[15225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:07:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:07:08.155596 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37447
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:08.532351 15238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:08.532796 15238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:08.532975 15238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:08.540855 15238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:08.541038 15238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:08.541155 15238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:08.541257 15238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:08.541349 15238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:08.552371 15238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:08.554167 15238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:08.555807 15238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:08.569396 15244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:08.569423 15243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:08.570132 15238 server_base.cc:1034] running on GCE node
W20260621 07:07:08.570243 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:08.571234 15238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:08.572947 15238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:08.574156 15238 hybrid_clock.cc:648] HybridClock initialized: now 1782025628574091 us; error 62 us; skew 500 ppm
I20260621 07:07:08.574560 15238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:08 dist-test-slave-hlq1 krb5kdc[15225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:08.578109 15238 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:08.579906 15238 webserver.cc:460] Webserver started at http://127.1.188.254:43409/ using document root <none> and password file <none>
I20260621 07:07:08.580706 15238 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:08.580853 15238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:08.581190 15238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:08.584679 15238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "51519e4e11a646629285dda435976515"
format_stamp: "Formatted at 2026-06-21 07:07:08 on dist-test-slave-hlq1"
server_key: "2e8054059a87b091ee48faa2604fb3a3"
server_key_iv: "eebcef41be5d72237f2e4840361cd83c"
server_key_version: "encryptionkey@0"
I20260621 07:07:08.585551 15238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "51519e4e11a646629285dda435976515"
format_stamp: "Formatted at 2026-06-21 07:07:08 on dist-test-slave-hlq1"
server_key: "2e8054059a87b091ee48faa2604fb3a3"
server_key_iv: "eebcef41be5d72237f2e4840361cd83c"
server_key_version: "encryptionkey@0"
I20260621 07:07:08.591534 15238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 07:07:08.595566 15253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:08.597208 15238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:07:08.597492 15238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "51519e4e11a646629285dda435976515"
format_stamp: "Formatted at 2026-06-21 07:07:08 on dist-test-slave-hlq1"
server_key: "2e8054059a87b091ee48faa2604fb3a3"
server_key_iv: "eebcef41be5d72237f2e4840361cd83c"
server_key_version: "encryptionkey@0"
I20260621 07:07:08.597752 15238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:08.624809 15238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:08.630095 15238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:08.630466 15238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:08.652465 15238 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:37447
I20260621 07:07:08.652467 15304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:37447 every 8 connection(s)
I20260621 07:07:08.654191 15238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:08.658952 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:08.662909 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15238
I20260621 07:07:08.663192 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:07:08.663996 1779 external_mini_cluster.cc:1428] Setting key 04aa7e2fb0ad9abbc462d0884a659989
I20260621 07:07:08.667117 15305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:08 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.253@KRBTEST.COM" created.
I20260621 07:07:08.695842 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.671292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37447 (local address 127.0.0.1:37840)
0621 07:07:08.672659 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.672698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:08.675830 (+ 3132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:08.676857 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:08.676867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:08.677784 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:08.678312 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:08.678341 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:08.680143 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:08.680151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:08.682120 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:08.682138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:08.683134 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:08.683147 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:08.683366 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:08.684187 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:08.684223 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:08.687769 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:08.691311 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:08.691322 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:08.691334 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:08.691837 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:08.692212 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:08.692216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:08.692218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:08.692382 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:08.693462 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:08.693469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:08.693696 (+ 227us) client_negotiation.cc:769] Sending connection context
0621 07:07:08.693959 (+ 263us) client_negotiation.cc:240] Negotiation successful
0621 07:07:08.694436 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"spinlock_wait_cycles":19712,"thread_start_us":829,"threads_started":1}
I20260621 07:07:08.695906 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.671285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.1.188.254:37447)
0621 07:07:08.672339 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:08.672362 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:08.672873 (+ 511us) server_negotiation.cc:407] Connection header received
0621 07:07:08.676277 (+ 3404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:08.676284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:08.676581 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:08.676765 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:08.678573 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:08.679975 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:08.682269 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:08.682876 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:08.683020 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:08.688051 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:08.688062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:08.688077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:08.688110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:08.691186 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:08.691954 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:08.691961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:08.691964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:08.692113 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:08.692512 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:08.692517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:08.692520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:08.693290 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:08.693513 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:08.694160 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 07:07:08.694659 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":760,"thread_start_us":444,"threads_started":1}
I20260621 07:07:08.699620 15305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:07:08.701062 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.700494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35184)
0621 07:07:08.700726 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.700880 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260621 07:07:08.701408 15255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
I20260621 07:07:08.703243 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.702836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35188)
0621 07:07:08.702981 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.703106 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 07:07:08.703508 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:46049: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
I20260621 07:07:08.725067 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:46049
--webserver_interface=127.1.188.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:07:08.752916 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } attempt: 1
I20260621 07:07:08.754441 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.753874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35200)
0621 07:07:08.754117 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.754276 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 07:07:08.755566 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.755137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35204)
0621 07:07:08.755283 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.755367 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:07:08.755781 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:46049: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
I20260621 07:07:08.824093 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } attempt: 2
I20260621 07:07:08.825661 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.825047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35220)
0621 07:07:08.825365 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.825483 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 07:07:08.826949 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.826518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35228)
0621 07:07:08.826695 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.826800 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 07:07:08.827244 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:46049: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
I20260621 07:07:08.919102 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } attempt: 3
I20260621 07:07:08.920907 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.920133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35236)
0621 07:07:08.920448 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.920596 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 07:07:08.922719 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:08.922191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35242)
0621 07:07:08.922429 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:08.922537 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260621 07:07:08.923177 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:46049: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
I20260621 07:07:09.066501 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } attempt: 4
I20260621 07:07:09.068194 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.067478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35244)
0621 07:07:09.067912 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.068020 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260621 07:07:09.069536 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.069106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35252)
0621 07:07:09.069284 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.069394 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260621 07:07:09.069898 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:46049: Network error: Client connection negotiation failed: client connection to 127.1.188.253:46049: connect: Connection refused (error 111)
W20260621 07:07:09.118607 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:09.118999 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:09.119204 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:09.127218 15312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:09.127380 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:09.127534 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:09.127652 15312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:09.127815 15312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:09.139232 15312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.188.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:09.141324 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:09.143265 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:09.156876 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:09.157042 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:09.157016 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:09.158181 15312 server_base.cc:1034] running on GCE node
I20260621 07:07:09.159104 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:09.160858 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:09.162082 15312 hybrid_clock.cc:648] HybridClock initialized: now 1782025629162039 us; error 29 us; skew 500 ppm
I20260621 07:07:09.162518 15312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:09.166172 15312 init.cc:377] Logged in from keytab as kudu/127.1.188.253@KRBTEST.COM (short username kudu)
I20260621 07:07:09.167904 15312 webserver.cc:460] Webserver started at http://127.1.188.253:44391/ using document root <none> and password file <none>
I20260621 07:07:09.168761 15312 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:09.168937 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:09.169291 15312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:09.172765 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data/instance:
uuid: "afb8536a9a4043e788c5c8e645d9172d"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "c37507ec700e7eaf5156116356928269"
server_key_iv: "cdc438630cd80c90b865056fd250e987"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.173676 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal/instance:
uuid: "afb8536a9a4043e788c5c8e645d9172d"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "c37507ec700e7eaf5156116356928269"
server_key_iv: "cdc438630cd80c90b865056fd250e987"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.179525 15312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 07:07:09.183310 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:09.184952 15312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:07:09.185151 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "afb8536a9a4043e788c5c8e645d9172d"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "c37507ec700e7eaf5156116356928269"
server_key_iv: "cdc438630cd80c90b865056fd250e987"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.185370 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:09.216403 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:09.221698 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:09.222044 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:09.244094 15312 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.253:46049
I20260621 07:07:09.244114 15378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.253:46049 every 8 connection(s)
I20260621 07:07:09.245879 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
I20260621 07:07:09.250445 15379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:09.252450 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15312
I20260621 07:07:09.252701 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-1/wal/instance
I20260621 07:07:09.253363 1779 external_mini_cluster.cc:1428] Setting key e95f2dc65a2454857b7c3b497cb8a843
I20260621 07:07:09.261759 15379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.252@KRBTEST.COM" created.
I20260621 07:07:09.286626 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.266022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.1.188.254:37447)
0621 07:07:09.266629 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.266635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.267993 (+ 1358us) server_negotiation.cc:407] Connection header received
0621 07:07:09.270250 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.270257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.270373 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.270514 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.272361 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.273173 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.274796 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.275241 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.275380 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.279272 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.279277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.279281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.279341 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.282769 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.283784 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.283789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.283791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.283952 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.284473 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.284477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.284479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.284801 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.284937 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.286053 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.286281 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":320,"threads_started":1}
I20260621 07:07:09.288620 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.266812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37447 (local address 127.0.0.1:37848)
0621 07:07:09.267771 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.267827 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.269915 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.270795 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.270805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.271481 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.272166 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.272185 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.273341 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.273346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.274649 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.274661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.275502 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.275509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.275762 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.276663 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.276688 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.279039 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.282919 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.282929 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.282950 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.283618 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.284065 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.284073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.284080 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.284345 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.284946 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.284960 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.285510 (+ 550us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.285894 (+ 384us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.286563 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":392,"threads_started":1}
I20260621 07:07:09.290784 15379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:07:09.314992 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.291956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35258)
0621 07:07:09.292170 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.292193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.292315 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.299739 (+ 7424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.299748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.300390 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.300628 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.300643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.302345 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.302354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.304063 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.304080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.305142 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.305148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.305384 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.306180 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.306210 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.308518 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.312373 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.312378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.312384 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.312833 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.313234 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.313246 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.313252 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.313416 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.314097 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.314102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.314228 (+ 126us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.314344 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.314625 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 07:07:09.315086 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.291988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.1.188.253:46049)
0621 07:07:09.299101 (+ 7113us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.299109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.299142 (+ 33us) server_negotiation.cc:407] Connection header received
0621 07:07:09.299204 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.299211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.299547 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.299700 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.300812 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.302153 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.304240 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.304968 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.305104 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.308747 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.308760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.308777 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.308807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.312233 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.312966 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.312974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.312977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.313105 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.313553 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.313557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.313559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.313956 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.314220 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.314514 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.314759 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6898,"thread_start_us":6713,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
I20260621 07:07:09.317958 15379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:07:09.319329 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.318845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:35954)
0621 07:07:09.319031 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.319172 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 07:07:09.319440 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:40727
--webserver_interface=127.1.188.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:09.319684 15328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.321384 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.321053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:35960)
0621 07:07:09.321203 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.321282 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 07:07:09.321630 15379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.369302 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } attempt: 5
I20260621 07:07:09.370992 15379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 1
I20260621 07:07:09.372287 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.371685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:35966)
0621 07:07:09.371909 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.372052 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 07:07:09.373536 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.373145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:35978)
0621 07:07:09.373325 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.373408 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:07:09.373844 15379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:07:09.386845 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.370430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.1.188.253:46049)
0621 07:07:09.370647 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.370656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.370685 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:07:09.370901 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.370908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.371051 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.371190 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.372413 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.373598 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.375508 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.376036 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.376142 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.380822 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.380832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.380838 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.380886 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.383951 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.384679 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.384691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.384694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.384829 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.385256 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.385262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.385264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.385592 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.385762 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.386219 (+ 457us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.386473 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:07:09.386878 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.370306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35274)
0621 07:07:09.370556 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.370586 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.370734 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.371284 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.371291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.371927 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.372224 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.372240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.373783 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.373789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.375295 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.375316 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.376243 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.376251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.376496 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.377324 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.377357 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.380604 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.384094 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.384101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.384106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.384545 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.384952 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.384959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.384963 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.385126 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.385720 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.385727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.385910 (+ 183us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.386098 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.386435 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 07:07:09.388181 15305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:07:09.389438 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.389009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:35988)
0621 07:07:09.389195 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.389283 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260621 07:07:09.390496 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.390160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36002)
0621 07:07:09.390290 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.390386 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260621 07:07:09.390743 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.442124 15379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 2
I20260621 07:07:09.442116 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 1
I20260621 07:07:09.443588 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.442956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36010)
0621 07:07:09.443190 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.443320 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:07:09.443661 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.443075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36008)
0621 07:07:09.443324 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.443444 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260621 07:07:09.444911 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.444471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36040)
0621 07:07:09.444710 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.444793 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260621 07:07:09.444933 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.444471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36028)
0621 07:07:09.444663 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.444763 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 07:07:09.445164 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
W20260621 07:07:09.445261 15379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.513463 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 2
I20260621 07:07:09.515134 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.514428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36054)
0621 07:07:09.514720 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.514869 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 07:07:09.516337 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.515934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36056)
0621 07:07:09.516089 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.516182 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 07:07:09.516566 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.536528 15379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 3
I20260621 07:07:09.538096 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.537525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36068)
0621 07:07:09.537789 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.537900 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 07:07:09.539556 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.539034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36078)
0621 07:07:09.539294 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.539376 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260621 07:07:09.539819 15379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.622843 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 3
I20260621 07:07:09.624607 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.623860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36080)
0621 07:07:09.624158 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.624320 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 07:07:09.626387 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.625809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36088)
0621 07:07:09.626110 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.626216 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260621 07:07:09.626755 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.683097 15379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 4
I20260621 07:07:09.684660 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.684046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36098)
0621 07:07:09.684295 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.684430 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 07:07:09.685788 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.685429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36114)
0621 07:07:09.685584 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.685668 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 07:07:09.686028 15379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
W20260621 07:07:09.715881 15387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:09.716279 15387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:09.716442 15387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:09.724325 15387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:09.724495 15387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:09.724596 15387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:09.724715 15387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:09.724783 15387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:09.735926 15387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:37447,127.1.188.253:46049,127.1.188.252:40727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.188.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:09.737607 15387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:09.739143 15387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:09.752599 15393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:09.752575 15392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:09.752987 15395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:09.753739 15387 server_base.cc:1034] running on GCE node
I20260621 07:07:09.754751 15387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:09.756618 15387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:09.757860 15387 hybrid_clock.cc:648] HybridClock initialized: now 1782025629757805 us; error 51 us; skew 500 ppm
I20260621 07:07:09.758270 15387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:09.761734 15387 init.cc:377] Logged in from keytab as kudu/127.1.188.252@KRBTEST.COM (short username kudu)
I20260621 07:07:09.763338 15387 webserver.cc:460] Webserver started at http://127.1.188.252:45539/ using document root <none> and password file <none>
I20260621 07:07:09.764192 15387 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:09.764338 15387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:09.764679 15387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:09.768134 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data/instance:
uuid: "94d5e5100a604ecf8eef2165cd34301f"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "e882bae457ebda13ca0bf3244a1c1bd7"
server_key_iv: "101d81700b25ae7737b6a0939a32bafa"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.769075 15387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal/instance:
uuid: "94d5e5100a604ecf8eef2165cd34301f"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "e882bae457ebda13ca0bf3244a1c1bd7"
server_key_iv: "101d81700b25ae7737b6a0939a32bafa"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.775126 15387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:07:09.779071 15402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:09.780812 15387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:09.781030 15387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "94d5e5100a604ecf8eef2165cd34301f"
format_stamp: "Formatted at 2026-06-21 07:07:09 on dist-test-slave-hlq1"
server_key: "e882bae457ebda13ca0bf3244a1c1bd7"
server_key_iv: "101d81700b25ae7737b6a0939a32bafa"
server_key_version: "encryptionkey@0"
I20260621 07:07:09.781272 15387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:09.794816 15387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:09.799484 15387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:09.799782 15387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:09.804075 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 4
I20260621 07:07:09.805303 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.804756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36126)
0621 07:07:09.804965 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.805071 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 07:07:09.806790 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.806211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36132)
0621 07:07:09.806452 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.806565 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 07:07:09.807094 15305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:40727: Network error: Client connection negotiation failed: client connection to 127.1.188.252:40727: connect: Connection refused (error 111)
I20260621 07:07:09.820642 15387 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.252:40727
I20260621 07:07:09.820667 15453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.252:40727 every 8 connection(s)
I20260621 07:07:09.822293 15387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
I20260621 07:07:09.826814 15454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:09.827658 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15387
I20260621 07:07:09.827899 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782025418728693-1779-0/minicluster-data/master-2/wal/instance
I20260621 07:07:09.828425 1779 external_mini_cluster.cc:1428] Setting key c2a890ce7dc1f039e021d90e603631fd
I20260621 07:07:09.837834 15454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:09.854856 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.830606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.1.188.252:40727)
0621 07:07:09.831612 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.831626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.831680 (+ 54us) server_negotiation.cc:407] Connection header received
0621 07:07:09.833644 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.833675 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.834204 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.834616 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.836732 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.838154 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.840087 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.840931 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.841140 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.846745 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.846754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.846766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.846807 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.850225 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.850967 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.850982 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.850998 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.851237 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.851662 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.851668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.851672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.852311 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.852585 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.853076 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.853510 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":440,"threads_started":1}
I20260621 07:07:09.858078 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.841365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.1.188.254:37447)
0621 07:07:09.842058 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.842066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.842652 (+ 586us) server_negotiation.cc:407] Connection header received
0621 07:07:09.842894 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.842898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.842988 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.843091 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.844750 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.845540 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.847324 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.848012 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.848165 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.851911 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.851916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.851920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.851959 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.855116 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.855919 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.855925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.855929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.856110 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.856473 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.856482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.856485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.856775 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.856894 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.857397 (+ 503us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.857612 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":290,"threads_started":1}
I20260621 07:07:09.858925 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.841377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:37447 (local address 127.0.0.1:37854)
0621 07:07:09.842475 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.842539 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.842749 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.843199 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.843207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.844182 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.844590 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.844605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.845704 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.845710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.847166 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.847181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.848229 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.848234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.848373 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.849120 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.849138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.851717 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.855283 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.855295 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.855303 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.855803 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.856215 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.856218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.856220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.856359 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.856893 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.856900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.857107 (+ 207us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.857307 (+ 200us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.857650 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":580,"threads_started":1}
I20260621 07:07:09.860522 15454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:07:09.874372 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.861440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:46049 (local address 127.0.0.1:35286)
0621 07:07:09.861773 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.861798 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.861926 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.862264 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.862268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.862657 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.862807 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.862815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.863889 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.863896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.864794 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.864806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.865532 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.865537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.865728 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.866295 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.866312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.868306 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.871572 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.871577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.871582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.872052 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.872495 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.872503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.872509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.872712 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.873322 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.873327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.873495 (+ 168us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.873682 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.873991 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260621 07:07:09.874660 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.861439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.1.188.253:46049)
0621 07:07:09.861659 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.861665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.861882 (+ 217us) server_negotiation.cc:407] Connection header received
0621 07:07:09.862059 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.862069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.862144 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.862248 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.862974 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.863762 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.864945 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.865389 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.865520 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.868516 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.868522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.868526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.868562 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.871377 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.872163 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.872170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.872174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.872366 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.872840 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.872849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.872854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.873213 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.873332 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.873795 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.874069 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:07:09.874895 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.858575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.1.188.254:37447)
0621 07:07:09.858767 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.858773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.858792 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:07:09.858961 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.858967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.859075 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.859172 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.860766 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.861784 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.863048 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.863551 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.863688 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.868527 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.868532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.868535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.868568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.871693 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.872412 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.872416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.872418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.872596 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.873094 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.873097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.873099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.873447 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.873759 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.874056 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.874299 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:07:09.875878 15454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:07:09.890885 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.876735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.1.188.252:40727)
0621 07:07:09.876931 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.876944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.877085 (+ 141us) server_negotiation.cc:407] Connection header received
0621 07:07:09.877242 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.877247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.877312 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.877402 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.878271 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.878979 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.880626 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.881135 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.881235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.884747 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.884754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.884763 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.884808 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.887940 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.888723 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.888727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.888729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.888838 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.889267 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.889272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.889275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.889647 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.889817 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.890189 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.890562 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":65}
I20260621 07:07:09.890885 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.876763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36144)
0621 07:07:09.877002 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.877018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.877141 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.877417 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.877421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.877906 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.878120 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.878135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.879106 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.879111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.880460 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.880474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.881291 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.881297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.881513 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.882144 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.882162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.884558 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.888085 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.888093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.888100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.888607 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.888942 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.888947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.888951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.889143 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:09.889785 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:09.889790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:09.889934 (+ 144us) client_negotiation.cc:769] Sending connection context
0621 07:07:09.890093 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 07:07:09.890401 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260621 07:07:09.893668 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.877321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.1.188.253:46049)
0621 07:07:09.877474 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.877478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.877490 (+ 12us) server_negotiation.cc:407] Connection header received
0621 07:07:09.877537 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.877541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.877595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.877666 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.879156 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.880262 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.882017 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.882492 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.882591 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.887606 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.887611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.887614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.887647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.890779 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.891684 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.891690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.891693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.891840 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.892221 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.892226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.892230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.892524 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.892818 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.893067 (+ 249us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.893299 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 07:07:09.897145 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: Bootstrap starting.
I20260621 07:07:09.898599 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:07:09.900969 15454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:09.902442 15454 log.cc:824] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:09.906767 15454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: No bootstrap required, opened a new log
I20260621 07:07:09.920578 15454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:09.921159 15454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:09.921316 15454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d5e5100a604ecf8eef2165cd34301f, State: Initialized, Role: FOLLOWER
I20260621 07:07:09.922418 15454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:09.930814 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.904541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.1.188.252:40727)
0621 07:07:09.904742 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.904749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.909360 (+ 4611us) server_negotiation.cc:407] Connection header received
0621 07:07:09.909524 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.909531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.909648 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.909781 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.912123 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.913413 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.915865 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.916633 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.916754 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.920968 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.920975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.920981 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.921042 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.925591 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.926969 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.926979 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.926998 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.927264 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.927973 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.927982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.927999 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.928538 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.928898 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.929855 (+ 957us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.930209 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:07:09.930992 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.904874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.1.188.253:46049)
0621 07:07:09.910472 (+ 5598us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.910478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.910492 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:07:09.910539 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.910542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.910599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.910721 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.913466 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.914988 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.917471 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.918250 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.918396 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.922312 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.922319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.922324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.922374 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.926607 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.927653 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.927660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.927665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.927847 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.928331 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.928335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.928338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.928744 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.928994 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.930248 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.930550 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5476}
I20260621 07:07:09.932912 15468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:09.933542 15468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:09.933874 15454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:09.936574 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.904296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.1.188.254:37447)
0621 07:07:09.904594 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.904602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.916116 (+ 11514us) server_negotiation.cc:407] Connection header received
0621 07:07:09.916269 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.916275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.916381 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.916489 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.919657 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.920936 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.923737 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.924486 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.924616 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.928491 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.928497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.928502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.928564 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.933615 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.934400 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.934406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.934410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.934567 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.934949 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.934960 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.934963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.935288 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:09.935621 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:09.935894 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:07:09.936144 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
W20260621 07:07:09.946264 15479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:07:09.946518 15479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:07:09.942082 15387 master_runner.cc:186] Error getting master registration for 127.1.188.254:37447: OK, instance_id { permanent_uuid: "51519e4e11a646629285dda435976515" instance_seqno: 1782025628646109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:07:09.956035 15387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:07:09.985262 15379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 5
Jun 21 07:07:09 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025629, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:07:10.002264 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.985977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36158)
0621 07:07:09.986208 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:09.986232 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:09.986346 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:09.986688 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:09.986693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:09.987161 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:09.987426 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.987445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.988980 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.988991 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.989997 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:09.990008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.991678 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.991684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:09.991912 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.992643 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:09.992669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:09.995654 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:09.999024 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:09.999030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:09.999034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:09.999629 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:10.000040 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:10.000045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:10.000050 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:10.000223 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:10.001053 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:10.001059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:10.001263 (+ 204us) client_negotiation.cc:769] Sending connection context
0621 07:07:10.001438 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 07:07:10.001808 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:07:10.003389 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:09.986045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.1.188.252:40727)
0621 07:07:09.986293 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:09.986300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:09.986322 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:07:09.986465 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:09.986472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:09.986562 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:09.986670 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:09.987587 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.988857 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.990123 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:09.991034 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:09.991179 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:09.995833 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:09.995839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:09.995843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:09.995880 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:09.998886 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:09.999746 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:09.999750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:09.999752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:09.999908 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:10.000338 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:10.000342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:10.000345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:10.000813 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:10.000979 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:10.002675 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0621 07:07:10.002963 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260621 07:07:10.010629 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: Bootstrap starting.
I20260621 07:07:10.016363 15379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:10.018219 15379 log.cc:824] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:10.022914 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: No bootstrap required, opened a new log
I20260621 07:07:10.036559 15379 raft_consensus.cc:348] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.037103 15379 raft_consensus.cc:374] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:10.037240 15379 raft_consensus.cc:729] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb8536a9a4043e788c5c8e645d9172d, State: Initialized, Role: FOLLOWER
I20260621 07:07:10.038064 15379 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.039176 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.039650 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.040967 15379 sys_catalog.cc:564] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:07:10.052060 15408 tablet.cc:2369] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:10.054807 15493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:07:10.054949 15493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:07:10.051487 15312 master_runner.cc:186] Error getting master registration for 127.1.188.254:37447: OK, instance_id { permanent_uuid: "51519e4e11a646629285dda435976515" instance_seqno: 1782025628646109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:07:10.060544 15312 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:07:10.084350 15305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } attempt: 5
I20260621 07:07:10.086401 15482 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:07:10.086752 15482 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.088653 15482 leader_election.cc:288] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51519e4e11a646629285dda435976515 (127.1.188.254:37447), 94d5e5100a604ecf8eef2165cd34301f (127.1.188.252:40727)
I20260621 07:07:10.089838 15429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb8536a9a4043e788c5c8e645d9172d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d5e5100a604ecf8eef2165cd34301f" is_pre_election: true
I20260621 07:07:10.090553 15429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb8536a9a4043e788c5c8e645d9172d in term 0.
I20260621 07:07:10.091238 15328 leader_election.cc:302] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d5e5100a604ecf8eef2165cd34301f, afb8536a9a4043e788c5c8e645d9172d; no voters:
I20260621 07:07:10.094251 15482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:07:10.094411 15482 raft_consensus.cc:482] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:07:10.094502 15482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:10.098217 15482 raft_consensus.cc:504] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.099838 15429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb8536a9a4043e788c5c8e645d9172d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94d5e5100a604ecf8eef2165cd34301f"
I20260621 07:07:10.100086 15429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 0 FOLLOWER]: Advancing to term 1
Jun 21 07:07:10 dist-test-slave-hlq1 krb5kdc[15225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025628, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:07:10.103484 15482 leader_election.cc:288] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [CANDIDATE]: Term 1 election: Requested vote from peers 51519e4e11a646629285dda435976515 (127.1.188.254:37447), 94d5e5100a604ecf8eef2165cd34301f (127.1.188.252:40727)
I20260621 07:07:10.104312 15429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb8536a9a4043e788c5c8e645d9172d in term 1.
I20260621 07:07:10.104805 15328 leader_election.cc:302] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d5e5100a604ecf8eef2165cd34301f, afb8536a9a4043e788c5c8e645d9172d; no voters:
I20260621 07:07:10.105203 15482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:10.113083 15482 raft_consensus.cc:686] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [term 1 LEADER]: Becoming Leader. State: Replica: afb8536a9a4043e788c5c8e645d9172d, State: Running, Role: LEADER
I20260621 07:07:10.113778 15482 consensus_queue.cc:237] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.118220 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:10.085225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:40727 (local address 127.0.0.1:36172)
0621 07:07:10.085633 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:10.085656 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:10.085794 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:10.087896 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:10.087904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:10.088504 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:10.088740 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:10.088753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:10.092108 (+ 3355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:10.092114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:10.093668 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:10.093684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:10.096094 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:10.096101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:10.096291 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:10.097200 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:10.097227 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:10.101829 (+ 4602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:10.108504 (+ 6675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:10.108510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:10.108538 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:10.109361 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:10.110138 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:10.110144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:10.110148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:10.110484 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:10.111657 (+ 1173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:10.111662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:10.111913 (+ 251us) client_negotiation.cc:769] Sending connection context
0621 07:07:10.112114 (+ 201us) client_negotiation.cc:240] Negotiation successful
0621 07:07:10.112909 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260621 07:07:10.119436 15457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:10.086116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.1.188.252:40727)
0621 07:07:10.086332 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:10.086340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:10.086362 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:07:10.086431 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:10.086436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:10.086529 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:10.086659 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:10.088981 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:10.090209 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:10.094927 (+ 4718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:10.095663 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:10.095797 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:10.102116 (+ 6319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:10.102126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:10.102133 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:10.102184 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:10.108278 (+ 6094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:10.109618 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:10.109652 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:10.109656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:10.109942 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:10.110674 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:10.110679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:10.110683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:10.111451 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:10.112708 (+ 1257us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:10.113353 (+ 645us) server_negotiation.cc:299] Negotiation successful
0621 07:07:10.117866 (+ 4513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:07:10.124184 15495 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb8536a9a4043e788c5c8e645d9172d. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.124657 15495 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:10.126180 15305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: Bootstrap starting.
I20260621 07:07:10.132732 15305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:10.134202 15305 log.cc:824] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:10.137924 15305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: No bootstrap required, opened a new log
I20260621 07:07:10.143800 15497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:10.146109 15497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:10.149438 15305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.149803 15305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:10.149926 15305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51519e4e11a646629285dda435976515, State: Initialized, Role: FOLLOWER
I20260621 07:07:10.150664 15305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } }
I20260621 07:07:10.159247 15429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [term 1 FOLLOWER]: Refusing update from remote peer afb8536a9a4043e788c5c8e645d9172d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:10.160488 15482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:10.163828 15305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:10.164160 15278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:10.164285 15280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb8536a9a4043e788c5c8e645d9172d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51519e4e11a646629285dda435976515" is_pre_election: true
I20260621 07:07:10.165225 15279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "afb8536a9a4043e788c5c8e645d9172d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51519e4e11a646629285dda435976515"
I20260621 07:07:10.168073 15278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [term 1 FOLLOWER]: Refusing update from remote peer afb8536a9a4043e788c5c8e645d9172d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:10.163376 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.170478 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.172081 15482 consensus_queue.cc:1035] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:10.189479 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb8536a9a4043e788c5c8e645d9172d. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.189913 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.196154 15468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb8536a9a4043e788c5c8e645d9172d. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.196491 15468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.201289 15468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.201555 15468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.201805 15499 mvcc.cc:205] Tried to move back new op lower bound from 7299176981115404288 to 7299176980951752704. Current Snapshot: MvccSnapshot[applied={T|T < 7299176981115404288}]
I20260621 07:07:10.216002 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.216373 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:10.216651 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.216950 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:10.218271 15497 catalog_manager.cc:1347] Generated new cluster ID: 5e7de13a6f8846d0b88afc36ffb9eada
I20260621 07:07:10.218417 15497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:10.218819 15500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afb8536a9a4043e788c5c8e645d9172d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51519e4e11a646629285dda435976515" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 37447 } } peers { permanent_uuid: "afb8536a9a4043e788c5c8e645d9172d" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 46049 } } peers { permanent_uuid: "94d5e5100a604ecf8eef2165cd34301f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 40727 } } }
I20260621 07:07:10.219097 15500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:07:10.279392 15497 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:07:10.281921 15497 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:10.298049 15497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afb8536a9a4043e788c5c8e645d9172d: Generated new TSK 0
I20260621 07:07:10.299516 15497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:07:10.320132 15520 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:07:10.948834 15479 catalog_manager.cc:1259] Loaded cluster ID: 5e7de13a6f8846d0b88afc36ffb9eada
I20260621 07:07:10.948956 15479 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: loading cluster ID for follower catalog manager: success
I20260621 07:07:10.951471 15479 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: acquiring CA information for follower catalog manager: success
I20260621 07:07:10.953246 15479 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94d5e5100a604ecf8eef2165cd34301f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:07:11.325672 15520 catalog_manager.cc:1259] Loaded cluster ID: 5e7de13a6f8846d0b88afc36ffb9eada
I20260621 07:07:11.325811 15520 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: loading cluster ID for follower catalog manager: success
I20260621 07:07:11.328919 15520 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: acquiring CA information for follower catalog manager: success
I20260621 07:07:11.331113 15520 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51519e4e11a646629285dda435976515: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:07:11.403949 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15238
I20260621 07:07:11.405601 15300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:11.507956 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15238
I20260621 07:07:11.527603 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15312
I20260621 07:07:11.528987 15374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:11.647923 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15312
W20260621 07:07:11.671873 15404 connection.cc:535] server connection from 127.0.0.1:36158 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:07:11.672202 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15387
I20260621 07:07:11.673623 15449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:11.801681 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15387
2026-06-21T07:07:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5047 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:07:11 dist-test-slave-hlq1 krb5kdc[15529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:07:11 dist-test-slave-hlq1 krb5kdc[15529](info): set up 2 sockets
Jun 21 07:07:11 dist-test-slave-hlq1 krb5kdc[15529](info): commencing operation
krb5kdc: starting...
W20260621 07:07:12.932349 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:07:13.195019 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:32945
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:13.547140 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:13.547503 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:13.547659 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:13.554917 15542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:13.555053 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:13.555163 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:13.555256 15542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:13.555335 15542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:13.565281 15542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:32945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:13.566644 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:13.567961 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:13.579797 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:13.579841 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:13.579879 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:13.581040 15542 server_base.cc:1034] running on GCE node
I20260621 07:07:13.581811 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:13.583288 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:13.584545 15542 hybrid_clock.cc:648] HybridClock initialized: now 1782025633584490 us; error 34 us; skew 500 ppm
I20260621 07:07:13.584897 15542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025633, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:13.588091 15542 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:13.589947 15542 webserver.cc:460] Webserver started at http://127.1.188.254:38019/ using document root <none> and password file <none>
I20260621 07:07:13.590806 15542 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:13.590955 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:13.591295 15542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:13.594338 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "48cc63227fc74043986f01d55c3395df"
format_stamp: "Formatted at 2026-06-21 07:07:13 on dist-test-slave-hlq1"
server_key: "89a22e101fee7f5bf804245798e55419"
server_key_iv: "957b7b471d100c34ae5c4737a483903b"
server_key_version: "encryptionkey@0"
I20260621 07:07:13.595084 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "48cc63227fc74043986f01d55c3395df"
format_stamp: "Formatted at 2026-06-21 07:07:13 on dist-test-slave-hlq1"
server_key: "89a22e101fee7f5bf804245798e55419"
server_key_iv: "957b7b471d100c34ae5c4737a483903b"
server_key_version: "encryptionkey@0"
I20260621 07:07:13.600373 15542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:07:13.604032 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:13.605641 15542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:07:13.605846 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "48cc63227fc74043986f01d55c3395df"
format_stamp: "Formatted at 2026-06-21 07:07:13 on dist-test-slave-hlq1"
server_key: "89a22e101fee7f5bf804245798e55419"
server_key_iv: "957b7b471d100c34ae5c4737a483903b"
server_key_version: "encryptionkey@0"
I20260621 07:07:13.606067 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:13.619787 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:13.625900 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:13.626224 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:13.646004 15542 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:32945
I20260621 07:07:13.646024 15608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:32945 every 8 connection(s)
I20260621 07:07:13.647454 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:13.651145 15609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:13.657902 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15542
I20260621 07:07:13.658159 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:07:13.659070 1779 external_mini_cluster.cc:1428] Setting key a388043a35c45571d22e0e7db2cf7e33
I20260621 07:07:13.662379 15609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df: Bootstrap starting.
I20260621 07:07:13.665887 15609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:13.666977 15609 log.cc:824] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:13.669695 15609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df: No bootstrap required, opened a new log
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:13.677414 15609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } }
I20260621 07:07:13.677829 15609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:13.677954 15609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc63227fc74043986f01d55c3395df, State: Initialized, Role: FOLLOWER
I20260621 07:07:13.678512 15609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } }
I20260621 07:07:13.678766 15609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:13.678908 15609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:13.679072 15609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:13.681591 15609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } }
I20260621 07:07:13.682075 15609 leader_election.cc:302] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48cc63227fc74043986f01d55c3395df; no voters:
I20260621 07:07:13.682324 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:13.660912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.1.188.254:32945)
0621 07:07:13.661677 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:13.661688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:13.661728 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:07:13.663506 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:13.663528 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:13.663913 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:13.664232 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:13.666210 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:13.667661 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:13.669034 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:13.669707 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:13.669891 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:13.674577 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:13.674588 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:13.674602 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:13.674645 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:13.677728 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:13.678600 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:13.678617 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:13.678622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:13.678873 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:13.679294 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:13.679299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:13.679302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:13.679973 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:13.680299 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:13.680752 (+ 453us) server_negotiation.cc:299] Negotiation successful
0621 07:07:13.681169 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":370,"threads_started":1}
I20260621 07:07:13.682803 15609 leader_election.cc:288] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:07:13.682904 15614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:13.684067 15614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [term 1 LEADER]: Becoming Leader. State: Replica: 48cc63227fc74043986f01d55c3395df, State: Running, Role: LEADER
I20260621 07:07:13.684688 15614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } }
I20260621 07:07:13.685184 15609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:13.686681 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:07:13.687633 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48cc63227fc74043986f01d55c3395df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } } }
I20260621 07:07:13.688048 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:13.694283 15616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48cc63227fc74043986f01d55c3395df. Latest consensus state: current_term: 1 leader_uuid: "48cc63227fc74043986f01d55c3395df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc63227fc74043986f01d55c3395df" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 32945 } } }
I20260621 07:07:13.694711 15616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:13.695899 15620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:13.701224 15620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:13.710304 15620 catalog_manager.cc:1347] Generated new cluster ID: d7eb1428ac174f1095dd90eee73fc570
I20260621 07:07:13.710444 15620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:13.719085 15620 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:07:13.720371 15620 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:13.728454 15620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48cc63227fc74043986f01d55c3395df: Generated new TSK 0
I20260621 07:07:13.729291 15620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:07:13.749058 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15542
I20260621 07:07:13.750200 15604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:13.842770 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15542
2026-06-21T07:07:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2030 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15638](info): set up 2 sockets
Jun 21 07:07:13 dist-test-slave-hlq1 krb5kdc[15638](info): commencing operation
krb5kdc: starting...
W20260621 07:07:14.932266 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:07:15 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:07:15.204523 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:15.536701 15651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:15.536998 15651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:15.537197 15651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:15.544548 15651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:15.544646 15651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:15.544751 15651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:15.544847 15651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:15.544925 15651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:15.554754 15651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:15.556144 15651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:15.557421 15651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:15.569025 15656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:15.569074 15657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:15.569128 15659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:15.569936 15651 server_base.cc:1034] running on GCE node
I20260621 07:07:15.570674 15651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:15.572098 15651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:15.573357 15651 hybrid_clock.cc:648] HybridClock initialized: now 1782025635573312 us; error 28 us; skew 500 ppm
I20260621 07:07:15.573719 15651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:15 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:15.577039 15651 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:15.578553 15651 webserver.cc:460] Webserver started at http://127.1.188.254:45301/ using document root <none> and password file <none>
I20260621 07:07:15.579241 15651 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:15.579384 15651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:15.579710 15651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:15.582736 15651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "47986044021f40aa99e9dddcfee75b56"
format_stamp: "Formatted at 2026-06-21 07:07:15 on dist-test-slave-hlq1"
server_key: "d7825af3cb7b2d1ffcf89c9a7107e5ad"
server_key_iv: "6d96ef676bc0f31b18a358b7ba567680"
server_key_version: "encryptionkey@0"
I20260621 07:07:15.583576 15651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "47986044021f40aa99e9dddcfee75b56"
format_stamp: "Formatted at 2026-06-21 07:07:15 on dist-test-slave-hlq1"
server_key: "d7825af3cb7b2d1ffcf89c9a7107e5ad"
server_key_iv: "6d96ef676bc0f31b18a358b7ba567680"
server_key_version: "encryptionkey@0"
I20260621 07:07:15.588784 15651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 07:07:15.592314 15666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:15.593739 15651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:07:15.593879 15651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "47986044021f40aa99e9dddcfee75b56"
format_stamp: "Formatted at 2026-06-21 07:07:15 on dist-test-slave-hlq1"
server_key: "d7825af3cb7b2d1ffcf89c9a7107e5ad"
server_key_iv: "6d96ef676bc0f31b18a358b7ba567680"
server_key_version: "encryptionkey@0"
I20260621 07:07:15.594045 15651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:15.616428 15651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:15.621858 15651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:15.622218 15651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:15.644579 15651 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:35683
I20260621 07:07:15.644614 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:35683 every 8 connection(s)
I20260621 07:07:15.645988 15651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:15.647990 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15651
I20260621 07:07:15.648206 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:07:15.649047 1779 external_mini_cluster.cc:1428] Setting key fda870d9e1510735d6d2b6b05b2dcf87
I20260621 07:07:15.650727 15718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 07:07:15 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:15.665602 15718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap starting.
I20260621 07:07:15.668839 15718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:15.670202 15718 log.cc:824] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:15.673342 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:15.651267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.1.188.254:35683)
0621 07:07:15.652174 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:15.652186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:15.652229 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:07:15.653937 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:15.653959 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:15.654372 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:15.654731 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:15.656594 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:15.657928 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:15.659826 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:15.660513 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:15.660653 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:15.665794 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:15.665804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:15.665819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:15.665857 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:15.668985 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:15.669895 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:15.669902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:15.669908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:15.670071 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:15.670488 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:15.670491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:15.670493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:15.671285 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:15.671688 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:15.671988 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 07:07:15.672366 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":445,"threads_started":1}
I20260621 07:07:15.673635 15718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: No bootstrap required, opened a new log
I20260621 07:07:15.681437 15718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:15.681986 15718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:15.682132 15718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Initialized, Role: FOLLOWER
I20260621 07:07:15.682797 15718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:15.683094 15718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:15.683270 15718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:15.683497 15718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:15.686062 15718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:15.686673 15718 leader_election.cc:302] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47986044021f40aa99e9dddcfee75b56; no voters:
I20260621 07:07:15.687296 15718 leader_election.cc:288] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:07:15.687484 15723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:15.688261 15723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 LEADER]: Becoming Leader. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Running, Role: LEADER
I20260621 07:07:15.688865 15723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:15.689760 15718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:15.691713 15725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47986044021f40aa99e9dddcfee75b56. Latest consensus state: current_term: 1 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:15.692090 15725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:15.692545 15724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:15.693953 15724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:15.695211 15730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:15.701817 15730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:15.712863 15730 catalog_manager.cc:1347] Generated new cluster ID: 44685f87f0634f1d9afb67e06713e97c
I20260621 07:07:15.712987 15730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:15.722410 15730 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:07:15.724457 15730 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:15.745107 15730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Generated new TSK 0
I20260621 07:07:15.746052 15730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:07:15.806581 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:16.168896 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:16.169163 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:16.169256 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:16.179273 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:16.179384 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:16.179572 15744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:07:16.189448 15744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:16.190799 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:16.192241 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:16.205628 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:16.205718 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:16.206084 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:16.206627 15744 server_base.cc:1034] running on GCE node
I20260621 07:07:16.207209 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:16.208218 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:16.209442 15744 hybrid_clock.cc:648] HybridClock initialized: now 1782025636209397 us; error 43 us; skew 500 ppm
I20260621 07:07:16.209764 15744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:16 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:16.212929 15744 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:07:16.214319 15744 webserver.cc:460] Webserver started at http://127.1.188.193:45707/ using document root <none> and password file <none>
I20260621 07:07:16.214972 15744 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:16.215070 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:16.215312 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:16.218187 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "9ac25d6252583d6392cf3b70f79b1bf4"
server_key_iv: "5a8bc5e14d70a10c52160d5a563faa33"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.218885 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "9ac25d6252583d6392cf3b70f79b1bf4"
server_key_iv: "5a8bc5e14d70a10c52160d5a563faa33"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.223887 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260621 07:07:16.227357 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:16.228716 15744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:07:16.228847 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "9ac25d6252583d6392cf3b70f79b1bf4"
server_key_iv: "5a8bc5e14d70a10c52160d5a563faa33"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.229002 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:16.246354 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:16.250735 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:16.250980 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:16.251971 15744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:16.253278 15744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:16.253382 15744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:16.253487 15744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:16.253533 15744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:16.288519 15744 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:40043
I20260621 07:07:16.288556 15871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:40043 every 8 connection(s)
I20260621 07:07:16.289898 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:16.292773 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15744
I20260621 07:07:16.292989 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:07:16.293756 1779 external_mini_cluster.cc:1428] Setting key b0e8774878721749b8e5115addb131de
Jun 21 07:07:16 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:16.313011 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:16.292724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:34421 (local address 127.1.188.254:35683)
0621 07:07:16.294982 (+ 2258us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:16.294988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:16.295010 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:07:16.297092 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:16.297098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:16.297201 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:16.297321 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:16.299074 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.299893 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.301463 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.301851 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.301978 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:16.305956 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:16.305965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:16.305974 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:16.306019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:16.309637 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:16.310367 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:16.310375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:16.310381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:16.310541 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:16.310959 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:16.310966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:16.310972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:16.311371 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:16.311552 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:16.312239 (+ 687us) server_negotiation.cc:299] Negotiation successful
0621 07:07:16.312494 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":2022,"threads_started":1}
I20260621 07:07:16.313738 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:16.293405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:34421)
0621 07:07:16.294468 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:16.294533 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:16.296781 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:16.297637 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:16.297646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:16.298238 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:16.298909 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:16.298926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.300057 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.300060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:16.301313 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:16.301325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.302086 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.302094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:16.302318 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:16.303077 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:16.303117 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:16.305733 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:16.309768 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:16.309775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:16.309786 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:16.310245 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:16.310667 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:16.310671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:16.310673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:16.310847 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:16.311532 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:16.311537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:16.311867 (+ 330us) client_negotiation.cc:769] Sending connection context
0621 07:07:16.312097 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 07:07:16.312482 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":484,"threads_started":1}
I20260621 07:07:16.315249 15872 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:16.315579 15872 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:16.316293 15872 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:07:16.318427 15683 ts_manager.cc:194] Registered new tserver with Master: d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:16.320070 15683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:34421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:07:16.347041 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:16.672984 15878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:16.673266 15878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:16.673360 15878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:16.680785 15878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:16.680915 15878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:16.681032 15878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:16.691772 15878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:16.693006 15878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:16.694255 15878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:16.707906 15883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:16.707857 15884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:16.707918 15886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:16.708549 15878 server_base.cc:1034] running on GCE node
I20260621 07:07:16.709084 15878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:16.710062 15878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:16.711280 15878 hybrid_clock.cc:648] HybridClock initialized: now 1782025636711239 us; error 34 us; skew 500 ppm
I20260621 07:07:16.711651 15878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:16 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:16.714715 15878 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:16.716214 15878 webserver.cc:460] Webserver started at http://127.1.188.194:33923/ using document root <none> and password file <none>
I20260621 07:07:16.716884 15878 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:16.717006 15878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:16.717310 15878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:16.720140 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "80ef1ded48e64b07824b085ed92c1580"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "b7979d84c3bc1f2ed674245f01a72c80"
server_key_iv: "f246e7a3d711d4a9b2b3f06a64c7217f"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.720885 15878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "80ef1ded48e64b07824b085ed92c1580"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "b7979d84c3bc1f2ed674245f01a72c80"
server_key_iv: "f246e7a3d711d4a9b2b3f06a64c7217f"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.725834 15878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:07:16.729224 15893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:16.730564 15878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:07:16.730723 15878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "80ef1ded48e64b07824b085ed92c1580"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "b7979d84c3bc1f2ed674245f01a72c80"
server_key_iv: "f246e7a3d711d4a9b2b3f06a64c7217f"
server_key_version: "encryptionkey@0"
I20260621 07:07:16.730899 15878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:16.748276 15878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:16.752547 15878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:16.752846 15878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:16.753798 15878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:16.755172 15878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:16.755244 15878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:16.755378 15878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:16.755483 15878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 07:07:16.789297 15878 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:42441
I20260621 07:07:16.789402 16005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:42441 every 8 connection(s)
I20260621 07:07:16.790692 15878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:16.799345 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 15878
I20260621 07:07:16.799607 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:07:16.800066 1779 external_mini_cluster.cc:1428] Setting key 9dbdb7aee9963504fc5e0e752b8d06aa
Jun 21 07:07:16 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025636, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:16.811681 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:16.793363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:45091 (local address 127.1.188.254:35683)
0621 07:07:16.793552 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:16.793557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:16.794717 (+ 1160us) server_negotiation.cc:407] Connection header received
0621 07:07:16.796600 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:16.796604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:16.796676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:16.796764 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:16.798443 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.799241 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.801182 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.801652 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.801788 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:16.805722 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:16.805728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:16.805732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:16.805765 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:16.808540 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:16.809318 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:16.809327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:16.809330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:16.809476 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:16.809938 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:16.809943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:16.809947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:16.810220 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:16.810374 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:16.811042 (+ 668us) server_negotiation.cc:299] Negotiation successful
0621 07:07:16.811294 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:07:16.812634 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:16.793832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:45091)
0621 07:07:16.794540 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:16.794581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:16.796311 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:16.796999 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:16.797007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:16.797650 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:16.798241 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:16.798258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.799418 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.799426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:16.801028 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:16.801041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:16.801881 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:16.801888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:16.802143 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:16.802885 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:16.802913 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:16.805464 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:16.808713 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:16.808723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:16.808742 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:16.809172 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:16.809623 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:16.809628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:16.809630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:16.809810 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:16.810345 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:16.810352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:16.810691 (+ 339us) client_negotiation.cc:769] Sending connection context
0621 07:07:16.810954 (+ 263us) client_negotiation.cc:240] Negotiation successful
0621 07:07:16.811516 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":287,"threads_started":1}
I20260621 07:07:16.814042 16006 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:16.814356 16006 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:16.815073 16006 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
I20260621 07:07:16.816591 15683 ts_manager.cc:194] Registered new tserver with Master: 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441)
I20260621 07:07:16.817924 15683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:45091
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:07:16.851290 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:17.173507 16011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:17.173766 16011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:17.173859 16011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:17.180882 16011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:17.180980 16011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:17.181089 16011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:17.190838 16011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:17.192143 16011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:17.193434 16011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:17.206727 16016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:17.206643 16017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:17.206768 16019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:17.207561 16011 server_base.cc:1034] running on GCE node
I20260621 07:07:17.208204 16011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:17.209272 16011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:17.210512 16011 hybrid_clock.cc:648] HybridClock initialized: now 1782025637210464 us; error 39 us; skew 500 ppm
I20260621 07:07:17.210822 16011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:17 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:17.220224 16011 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:17.221609 16011 webserver.cc:460] Webserver started at http://127.1.188.195:37711/ using document root <none> and password file <none>
I20260621 07:07:17.222254 16011 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:17.222350 16011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:17.222584 16011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:17.225438 16011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "979ab4123e4640139993d5c13d890bf9"
format_stamp: "Formatted at 2026-06-21 07:07:17 on dist-test-slave-hlq1"
server_key: "985b2aa2edbce0dc5535218ce5269fe4"
server_key_iv: "8d279a3090a60b76ff559f1264517dc3"
server_key_version: "encryptionkey@0"
I20260621 07:07:17.226107 16011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "979ab4123e4640139993d5c13d890bf9"
format_stamp: "Formatted at 2026-06-21 07:07:17 on dist-test-slave-hlq1"
server_key: "985b2aa2edbce0dc5535218ce5269fe4"
server_key_iv: "8d279a3090a60b76ff559f1264517dc3"
server_key_version: "encryptionkey@0"
I20260621 07:07:17.231014 16011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:07:17.234606 16026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:17.236008 16011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:17.236187 16011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "979ab4123e4640139993d5c13d890bf9"
format_stamp: "Formatted at 2026-06-21 07:07:17 on dist-test-slave-hlq1"
server_key: "985b2aa2edbce0dc5535218ce5269fe4"
server_key_iv: "8d279a3090a60b76ff559f1264517dc3"
server_key_version: "encryptionkey@0"
I20260621 07:07:17.236393 16011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:17.253790 16011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:17.258100 16011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:17.258440 16011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:17.259934 16011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:17.261350 16011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:17.261451 16011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:17.261617 16011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:17.261693 16011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:17.297268 16011 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:35227
I20260621 07:07:17.297436 16138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:35227 every 8 connection(s)
I20260621 07:07:17.298817 16011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:17.303499 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16011
I20260621 07:07:17.303697 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:07:17.304265 1779 external_mini_cluster.cc:1428] Setting key b2710088c796caf67f1f0ba6cf0cb5ce
Jun 21 07:07:17 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025637, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:17.319469 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.301575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:35949 (local address 127.1.188.254:35683)
0621 07:07:17.301802 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.301808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.302825 (+ 1017us) server_negotiation.cc:407] Connection header received
0621 07:07:17.305261 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.305266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.305382 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.305492 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:17.307109 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.307944 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.309425 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.309821 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.309944 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.313435 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:17.313440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:17.313443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:17.313476 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:17.316298 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.316931 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.316935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.316937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.317063 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.317439 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.317442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.317445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.317777 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:17.317939 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.318819 (+ 880us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.319049 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:07:17.320375 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.301973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:35949)
0621 07:07:17.302677 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.302717 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.305005 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.305770 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.305781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.306415 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:17.306951 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.306967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.308087 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.308092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.309283 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.309294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.310048 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.310055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.310278 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.310970 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:17.310991 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:17.313209 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:17.316419 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:17.316425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:17.316438 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:17.316832 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:17.317174 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:17.317177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:17.317180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:17.317346 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:17.317981 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:17.317991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:17.318420 (+ 429us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.318739 (+ 319us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.319237 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":268,"threads_started":1}
I20260621 07:07:17.321609 16139 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:17.321911 16139 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:17.322628 16139 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
I20260621 07:07:17.323009 15872 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:17.324182 15682 ts_manager.cc:194] Registered new tserver with Master: 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227)
I20260621 07:07:17.325069 15682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:35949
I20260621 07:07:17.331668 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:07:17.350469 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.336439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.1.188.254:35683)
0621 07:07:17.336664 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.336681 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.336946 (+ 265us) server_negotiation.cc:407] Connection header received
0621 07:07:17.337075 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.337078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.337151 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.337246 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:17.338578 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.339336 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.340809 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.341311 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.341412 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.343799 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:17.343804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:17.343807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:17.343837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:17.346635 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.347828 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.347870 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.347877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.348060 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.348579 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.348586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.348591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.349068 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:17.349328 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.349795 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.350073 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:07:17.356057 15682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:07:17.367187 15682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:07:17.405696 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.383889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.1.188.193:40043)
0621 07:07:17.395718 (+ 11829us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.395730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.395762 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:07:17.395879 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.395891 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.396182 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.396394 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:17.397911 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.399839 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.402954 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.404533 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.404735 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.404970 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.405178 (+ 208us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.405344 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11608,"thread_start_us":12684,"threads_started":1}
I20260621 07:07:17.405710 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.383785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:48970)
0621 07:07:17.393880 (+ 10095us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.393901 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.394018 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.396396 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.396405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.396432 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:17.396823 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.396837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.400024 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.400031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.402741 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.402760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.404735 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.404741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.404925 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.404979 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.405137 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.405385 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9952,"thread_start_us":9464,"threads_started":1}
I20260621 07:07:17.409845 15807 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ae68311d140b58c0b86179f1393e1 (DEFAULT_TABLE table=test-table [id=a64911b15417436da6f466e8e1235edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:17.411343 15807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8ae68311d140b58c0b86179f1393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:17.412406 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.382562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:35227 (local address 127.0.0.1:59638)
0621 07:07:17.393524 (+ 10962us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.393548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.393701 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.403562 (+ 9861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.403571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.403623 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:17.404125 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.404139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.407507 (+ 3368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.407512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.409609 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.409623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.411388 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.411526 (+ 138us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.411716 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.411740 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.411868 (+ 128us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.412015 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10787,"thread_start_us":1375,"threads_started":1}
I20260621 07:07:17.412446 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.383308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.1.188.195:35227)
0621 07:07:17.399679 (+ 16371us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.399691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.399734 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:07:17.399853 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.399864 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.403250 (+ 3386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.404391 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:17.404735 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.407309 (+ 2574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.409782 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.411182 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.411324 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.411592 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.411936 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.412087 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16121,"thread_start_us":19620,"threads_started":1}
I20260621 07:07:17.415243 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.382103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:42441 (local address 127.0.0.1:52878)
0621 07:07:17.383301 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.383349 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.389763 (+ 6414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.408400 (+ 18637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.408405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.408436 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:17.408781 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.408790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.410279 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.410284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.412379 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.412396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.414452 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.414458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.414632 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.414670 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.414775 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.414901 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":11488,"threads_started":1}
I20260621 07:07:17.415525 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.382366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.1.188.194:42441)
0621 07:07:17.407947 (+ 25581us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.407954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.407977 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:07:17.408056 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.408061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.408221 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.408417 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:17.408947 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.410124 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.412927 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.414249 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.414406 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.414590 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.414827 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.414979 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25429,"thread_start_us":25297,"threads_started":1}
I20260621 07:07:17.416741 16074 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ae68311d140b58c0b86179f1393e1 (DEFAULT_TABLE table=test-table [id=a64911b15417436da6f466e8e1235edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:17.418196 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8ae68311d140b58c0b86179f1393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:17.418216 15941 tablet_service.cc:1467] Processing CreateTablet for tablet ff8ae68311d140b58c0b86179f1393e1 (DEFAULT_TABLE table=test-table [id=a64911b15417436da6f466e8e1235edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:17.419488 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8ae68311d140b58c0b86179f1393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:17.427220 16157 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Bootstrap starting.
I20260621 07:07:17.430943 16158 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Bootstrap starting.
I20260621 07:07:17.431005 16157 tablet_bootstrap.cc:654] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:17.432650 16157 log.cc:824] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:17.434202 16159 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Bootstrap starting.
I20260621 07:07:17.434507 16158 tablet_bootstrap.cc:654] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:17.435771 16158 log.cc:824] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:17.436381 16157 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: No bootstrap required, opened a new log
I20260621 07:07:17.436664 16157 ts_tablet_manager.cc:1397] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260621 07:07:17.438405 16159 tablet_bootstrap.cc:654] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:17.438902 16158 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: No bootstrap required, opened a new log
I20260621 07:07:17.439165 16158 ts_tablet_manager.cc:1397] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 07:07:17.439508 16159 log.cc:824] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:17.442155 16159 tablet_bootstrap.cc:492] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: No bootstrap required, opened a new log
I20260621 07:07:17.442415 16159 ts_tablet_manager.cc:1397] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 07:07:17.447444 16157 raft_consensus.cc:348] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.447889 16157 raft_consensus.cc:374] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:17.447980 16157 raft_consensus.cc:729] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4676d8d8f4341808a988ca9a7a5ccbd, State: Initialized, Role: FOLLOWER
I20260621 07:07:17.448696 16157 consensus_queue.cc:260] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.449628 16158 raft_consensus.cc:348] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.450037 16158 raft_consensus.cc:374] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:17.450138 16158 raft_consensus.cc:729] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ab4123e4640139993d5c13d890bf9, State: Initialized, Role: FOLLOWER
I20260621 07:07:17.450150 16159 raft_consensus.cc:348] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.450500 16157 ts_tablet_manager.cc:1428] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260621 07:07:17.450539 16159 raft_consensus.cc:374] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:17.450649 16159 raft_consensus.cc:729] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ef1ded48e64b07824b085ed92c1580, State: Initialized, Role: FOLLOWER
I20260621 07:07:17.450706 16158 consensus_queue.cc:260] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.451182 16159 consensus_queue.cc:260] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.451962 16139 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:17.452347 16158 ts_tablet_manager.cc:1428] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260621 07:07:17.455371 16006 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:17.455389 16159 ts_tablet_manager.cc:1428] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260621 07:07:17.542431 15873 tablet.cc:2369] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:17.542814 16007 tablet.cc:2369] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:17.550410 16140 tablet.cc:2369] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:07:17.556120 16165 raft_consensus.cc:482] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:07:17.556532 16165 raft_consensus.cc:504] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.558483 16165 leader_election.cc:288] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227), d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:17.565349 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.559344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:52489 (local address 127.1.188.193:40043)
0621 07:07:17.559585 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.559592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.559896 (+ 304us) server_negotiation.cc:407] Connection header received
0621 07:07:17.560090 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.560101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.560159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.560258 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:17.560916 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.561611 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.563570 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.564359 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.564534 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.564613 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.564968 (+ 355us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.565093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 07:07:17.565498 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.558729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:37349 (local address 127.1.188.195:35227)
0621 07:07:17.558981 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.558990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.559604 (+ 614us) server_negotiation.cc:407] Connection header received
0621 07:07:17.559977 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.559981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.560047 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.560149 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:17.560794 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.561601 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.563629 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.564482 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.564680 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.564764 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.565080 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.565225 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:07:17.565513 16167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.558988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.1.188.194:52489)
0621 07:07:17.559795 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.559808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.559958 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.560277 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.560281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.560313 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:17.560764 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.560772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.561752 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.561757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.563374 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.563386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.564577 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.564583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.564812 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.564837 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.564943 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.565069 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":92,"thread_start_us":493,"threads_started":1}
I20260621 07:07:17.565630 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.558627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:35227 (local address 127.1.188.194:37349)
0621 07:07:17.559472 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.559503 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:17.559792 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:17.560179 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:17.560184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:17.560212 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:17.560629 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.560640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.561744 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.561752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.563478 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:17.563491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.564726 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.564734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:17.564909 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.564933 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:07:17.565042 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 07:07:17.565172 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":24,"thread_start_us":354,"threads_started":1}
I20260621 07:07:17.566745 16094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8ae68311d140b58c0b86179f1393e1" candidate_uuid: "80ef1ded48e64b07824b085ed92c1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979ab4123e4640139993d5c13d890bf9" is_pre_election: true
I20260621 07:07:17.566776 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8ae68311d140b58c0b86179f1393e1" candidate_uuid: "80ef1ded48e64b07824b085ed92c1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" is_pre_election: true
I20260621 07:07:17.567378 15827 raft_consensus.cc:2454] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ef1ded48e64b07824b085ed92c1580 in term 0.
I20260621 07:07:17.567378 16094 raft_consensus.cc:2454] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ef1ded48e64b07824b085ed92c1580 in term 0.
I20260621 07:07:17.568210 15895 leader_election.cc:302] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ef1ded48e64b07824b085ed92c1580, 979ab4123e4640139993d5c13d890bf9; no voters:
I20260621 07:07:17.568572 16165 raft_consensus.cc:2781] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:07:17.568713 16165 raft_consensus.cc:482] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:07:17.568835 16165 raft_consensus.cc:3037] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:17.570979 16165 raft_consensus.cc:504] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.571661 16165 leader_election.cc:288] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [CANDIDATE]: Term 1 election: Requested vote from peers 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227), d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:17.572129 15827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8ae68311d140b58c0b86179f1393e1" candidate_uuid: "80ef1ded48e64b07824b085ed92c1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
I20260621 07:07:17.572140 16094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8ae68311d140b58c0b86179f1393e1" candidate_uuid: "80ef1ded48e64b07824b085ed92c1580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "979ab4123e4640139993d5c13d890bf9"
I20260621 07:07:17.572347 15827 raft_consensus.cc:3037] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:17.572340 16094 raft_consensus.cc:3037] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:17.574478 15827 raft_consensus.cc:2454] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ef1ded48e64b07824b085ed92c1580 in term 1.
I20260621 07:07:17.574546 16094 raft_consensus.cc:2454] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ef1ded48e64b07824b085ed92c1580 in term 1.
I20260621 07:07:17.574939 15896 leader_election.cc:302] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ef1ded48e64b07824b085ed92c1580, d4676d8d8f4341808a988ca9a7a5ccbd; no voters:
I20260621 07:07:17.575251 16165 raft_consensus.cc:2781] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:17.575762 16165 raft_consensus.cc:686] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 1 LEADER]: Becoming Leader. State: Replica: 80ef1ded48e64b07824b085ed92c1580, State: Running, Role: LEADER
I20260621 07:07:17.576292 16165 consensus_queue.cc:237] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } }
I20260621 07:07:17.580819 15683 catalog_manager.cc:5495] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194). New cstate: current_term: 1 leader_uuid: "80ef1ded48e64b07824b085ed92c1580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } health_report { overall_health: UNKNOWN } } }
I20260621 07:07:17.599071 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.593736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.1.188.194:42441)
0621 07:07:17.593948 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.593956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.593978 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:07:17.594077 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.594081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.594158 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.594304 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:07:17.594673 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.595392 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.596815 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.597261 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.597391 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.597961 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:07:17.598496 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:07:17.598688 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.598737 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.598817 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 07:07:17.607852 15827 raft_consensus.cc:1261] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Refusing update from remote peer 80ef1ded48e64b07824b085ed92c1580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:17.607852 16094 raft_consensus.cc:1261] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Refusing update from remote peer 80ef1ded48e64b07824b085ed92c1580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:17.608930 16168 consensus_queue.cc:1035] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:17.609324 16165 consensus_queue.cc:1035] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:17.621070 16173 mvcc.cc:205] Tried to move back new op lower bound from 7299177011631247360 to 7299177011516186624. Current Snapshot: MvccSnapshot[applied={T|T < 7299177011631247360}]
I20260621 07:07:17.621708 16172 mvcc.cc:205] Tried to move back new op lower bound from 7299177011631247360 to 7299177011516186624. Current Snapshot: MvccSnapshot[applied={T|T < 7299177011631247360}]
I20260621 07:07:17.640584 15683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:17.640919 15683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:17.645094 15683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Sending DeleteTablet for 3 replicas of tablet ff8ae68311d140b58c0b86179f1393e1
I20260621 07:07:17.646188 15941 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ae68311d140b58c0b86179f1393e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:52878
I20260621 07:07:17.646407 15807 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ae68311d140b58c0b86179f1393e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:48970
I20260621 07:07:17.646800 16074 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8ae68311d140b58c0b86179f1393e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:17 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59638
I20260621 07:07:17.648171 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15744
I20260621 07:07:17.666203 16182 tablet_replica.cc:330] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: stopping tablet replica
I20260621 07:07:17.666869 16182 raft_consensus.cc:2229] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:17.667598 16182 raft_consensus.cc:2258] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:17.668718 16181 tablet_replica.cc:330] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: stopping tablet replica
I20260621 07:07:17.669242 16181 raft_consensus.cc:2229] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:07:17.670145 16181 raft_consensus.cc:2258] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:17.670966 16183 tablet_replica.cc:330] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: stopping tablet replica
I20260621 07:07:17.671334 16183 raft_consensus.cc:2229] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:17.673681 16182 ts_tablet_manager.cc:1905] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:17.675958 16181 ts_tablet_manager.cc:1905] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 07:07:17 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:07:17.683681 16181 ts_tablet_manager.cc:1918] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:17.683903 16181 log.cc:1196] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/ff8ae68311d140b58c0b86179f1393e1
I20260621 07:07:17.684469 16181 ts_tablet_manager.cc:1939] T ff8ae68311d140b58c0b86179f1393e1 P 80ef1ded48e64b07824b085ed92c1580: Deleting consensus metadata
I20260621 07:07:17.686148 16156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.649190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.1.188.193:40043)
0621 07:07:17.649394 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.649402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.664403 (+ 15001us) server_negotiation.cc:407] Connection header received
0621 07:07:17.664666 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.664681 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.664768 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.664880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:17.668955 (+ 4075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.670061 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.672252 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.672909 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.673002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.678113 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:17.678125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:17.678139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:17.678182 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:17.682006 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.682944 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.682956 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.682962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.683151 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.683781 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.683787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.683911 (+ 124us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.684486 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:17.684903 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.685476 (+ 573us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.685742 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 07:07:17.686452 15670 catalog_manager.cc:4841] TS 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441): tablet ff8ae68311d140b58c0b86179f1393e1 (table test-table [id=a64911b15417436da6f466e8e1235edf]) successfully deleted
I20260621 07:07:17.686697 16182 ts_tablet_manager.cc:1918] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:17.686872 16182 log.cc:1196] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/ff8ae68311d140b58c0b86179f1393e1
I20260621 07:07:17.687332 16183 raft_consensus.cc:2258] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:17.687386 16182 ts_tablet_manager.cc:1939] T ff8ae68311d140b58c0b86179f1393e1 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting consensus metadata
I20260621 07:07:17.688129 15867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:17.689800 16183 ts_tablet_manager.cc:1905] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:17.697391 16183 ts_tablet_manager.cc:1918] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:17.697577 16183 log.cc:1196] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/ff8ae68311d140b58c0b86179f1393e1
I20260621 07:07:17.698208 16183 ts_tablet_manager.cc:1939] T ff8ae68311d140b58c0b86179f1393e1 P 979ab4123e4640139993d5c13d890bf9: Deleting consensus metadata
I20260621 07:07:17.700160 15668 catalog_manager.cc:4841] TS 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227): tablet ff8ae68311d140b58c0b86179f1393e1 (table test-table [id=a64911b15417436da6f466e8e1235edf]) successfully deleted
I20260621 07:07:17.781934 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15744
W20260621 07:07:17.800626 15669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.188.193:40043 (error 108)
I20260621 07:07:17.801383 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15878
I20260621 07:07:17.803817 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.803252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:48990)
0621 07:07:17.803520 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.803642 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260621 07:07:17.804069 15669 catalog_manager.cc:4584] TS d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff8ae68311d140b58c0b86179f1393e1: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Jun 21 07:07:17 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:07:17.818095 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.802751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.1.188.194:42441)
0621 07:07:17.802927 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.802935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.802955 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:07:17.803023 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.803027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.803142 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.803240 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:17.804906 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.805749 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.807086 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.807695 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.807833 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.812090 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:17.812103 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:17.812117 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:17.812162 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:17.815207 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.815926 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.815936 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.815938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.816085 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.816482 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.816486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.816489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.816836 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:17.817160 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.817473 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.817709 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:07:17.818940 16001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:17.857811 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.857224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:48996)
0621 07:07:17.857462 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.857582 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 07:07:17.858927 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.858529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49012)
0621 07:07:17.858693 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.858776 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 07:07:17.919770 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.919224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49026)
0621 07:07:17.919474 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.919579 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 07:07:17.920840 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.920465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49030)
0621 07:07:17.920611 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:17.920702 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:07:17.934523 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15878
I20260621 07:07:17.949703 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16011
Jun 21 07:07:17 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:07:17.965303 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:17.950529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.1.188.195:35227)
0621 07:07:17.950713 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:17.950718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:17.950736 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:07:17.950903 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:17.950908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:17.950986 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:17.951074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:17.952471 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.953416 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.954796 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:17.955235 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:17.955338 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:17.959754 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:17.959768 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:17.959785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:17.959821 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:17.962462 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.963230 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.963238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.963240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.963386 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:17.963904 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:17.963908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:17.963910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:17.964310 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:17.964535 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:17.964825 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 07:07:17.965006 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:07:17.965940 16134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:18.001573 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:18.001059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49034)
0621 07:07:18.001291 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:18.001396 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 07:07:18.002817 16153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:18.002404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49044)
0621 07:07:18.002620 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:18.002692 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:40043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260621 07:07:18.056173 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16011
I20260621 07:07:18.072832 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 15651
I20260621 07:07:18.074013 15713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:18.190973 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 15651
I20260621 07:07:18.212849 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--webserver_interface=127.1.188.254
--webserver_port=45301
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:18.540746 16190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:18.541035 16190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:18.541201 16190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:18.548477 16190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:18.548588 16190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:18.548700 16190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:18.548800 16190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:18.548861 16190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:18.558766 16190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=45301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:18.560195 16190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:18.561522 16190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:18.585182 16196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:18.585219 16199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:18.585182 16197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:18.586338 16190 server_base.cc:1034] running on GCE node
I20260621 07:07:18.587289 16190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:18.589023 16190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:18.590230 16190 hybrid_clock.cc:648] HybridClock initialized: now 1782025638590195 us; error 35 us; skew 500 ppm
I20260621 07:07:18.590628 16190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:18 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025638, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:18.594460 16190 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:18.596285 16190 webserver.cc:460] Webserver started at http://127.1.188.254:45301/ using document root <none> and password file <none>
I20260621 07:07:18.597084 16190 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:18.597203 16190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:18.604090 16190 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:07:18.607295 16206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:18.608747 16190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:18.608945 16190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "47986044021f40aa99e9dddcfee75b56"
format_stamp: "Formatted at 2026-06-21 07:07:15 on dist-test-slave-hlq1"
server_key: "d7825af3cb7b2d1ffcf89c9a7107e5ad"
server_key_iv: "6d96ef676bc0f31b18a358b7ba567680"
server_key_version: "encryptionkey@0"
I20260621 07:07:18.610098 16190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:18.626319 16190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:18.631234 16190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:18.631608 16190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:18.653563 16190 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:35683
I20260621 07:07:18.653584 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:35683 every 8 connection(s)
I20260621 07:07:18.655257 16190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:18.656605 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16190
I20260621 07:07:18.657459 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:40043
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=45707
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:07:18.662603 16258 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:07:18.667210 16258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap starting.
I20260621 07:07:18.697430 16258 log.cc:824] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:18.712814 16258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:07:18.713451 16258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap complete.
I20260621 07:07:18.723054 16258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:18.723999 16258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Initialized, Role: FOLLOWER
I20260621 07:07:18.724814 16258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:18.725083 16258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:18.725234 16258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:18.725415 16258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:07:18.729436 16258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:18.729969 16258 leader_election.cc:302] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47986044021f40aa99e9dddcfee75b56; no voters:
I20260621 07:07:18.730530 16258 leader_election.cc:288] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:07:18.730721 16262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:07:18.731851 16262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 LEADER]: Becoming Leader. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Running, Role: LEADER
I20260621 07:07:18.732450 16262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:18.732771 16258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:18.735641 16263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:18.736001 16263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:18.735069 16264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47986044021f40aa99e9dddcfee75b56. Latest consensus state: current_term: 2 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:18.736649 16264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:18.744831 16269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:18.753001 16269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:18.754438 16269 catalog_manager.cc:1259] Loaded cluster ID: 44685f87f0634f1d9afb67e06713e97c
I20260621 07:07:18.754537 16269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:18.757838 16269 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 07:07:18.775723 16269 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 07:07:18.790477 16280 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:07:18.792372 16269 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:18.793913 16269 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Loaded TSK: 0
I20260621 07:07:18.795061 16269 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:07:19.054395 16260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:19.054646 16260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:19.054790 16260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:19.061983 16260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:19.062081 16260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:19.062254 16260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:07:19.071785 16260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:40043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=45707
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:19.073027 16260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:19.074326 16260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:19.088150 16285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:19.088639 16286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:19.089246 16288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:19.089696 16260 server_base.cc:1034] running on GCE node
I20260621 07:07:19.090272 16260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:19.091249 16260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:19.092471 16260 hybrid_clock.cc:648] HybridClock initialized: now 1782025639092429 us; error 35 us; skew 500 ppm
I20260621 07:07:19.092787 16260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:19 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:19.095983 16260 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:07:19.097472 16260 webserver.cc:460] Webserver started at http://127.1.188.193:45707/ using document root <none> and password file <none>
I20260621 07:07:19.098145 16260 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:19.098280 16260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:19.103868 16260 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:07:19.106770 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.108358 16260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:19.108534 16260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "9ac25d6252583d6392cf3b70f79b1bf4"
server_key_iv: "5a8bc5e14d70a10c52160d5a563faa33"
server_key_version: "encryptionkey@0"
I20260621 07:07:19.109504 16260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:19.140874 16260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:19.145582 16260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:19.145874 16260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:19.146914 16260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:19.148384 16260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:19.148478 16260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.148629 16260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:19.148725 16260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.184991 16260 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:40043
I20260621 07:07:19.185081 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:40043 every 8 connection(s)
I20260621 07:07:19.186506 16260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:19.195696 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16260
I20260621 07:07:19.196337 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:42441
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=33923
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:07:19 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:19.214155 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:19.190043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:36845)
0621 07:07:19.191234 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:19.191297 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:19.193965 (+ 2668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:19.195439 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:19.195451 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:19.196194 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:19.196971 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:19.196995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.199525 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.199531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:19.201654 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:19.201670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.203237 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.203244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:19.203556 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:19.204277 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:19.204303 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:19.206407 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:19.209761 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:19.209769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:19.209783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:19.210181 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:19.210605 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:19.210612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:19.210616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:19.210801 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:19.211583 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:19.211591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:19.211920 (+ 329us) client_negotiation.cc:769] Sending connection context
0621 07:07:19.212181 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 07:07:19.212796 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":612,"threads_started":1}
I20260621 07:07:19.214156 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:19.189920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:36845 (local address 127.1.188.254:35683)
0621 07:07:19.190797 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:19.190813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:19.191472 (+ 659us) server_negotiation.cc:407] Connection header received
0621 07:07:19.194514 (+ 3042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:19.194527 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:19.194841 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:19.195232 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:19.197257 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.199301 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.201831 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.202980 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.203174 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:19.206675 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:19.206684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:19.206698 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:19.206733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:19.209606 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:19.210330 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:19.210337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:19.210341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:19.210492 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:19.210927 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:19.210932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:19.210934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:19.211444 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:19.211651 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:19.212392 (+ 741us) server_negotiation.cc:299] Negotiation successful
0621 07:07:19.212796 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":382,"threads_started":1}
I20260621 07:07:19.217402 16408 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:19.217752 16408 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:19.218708 16408 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
I20260621 07:07:19.221369 16223 ts_manager.cc:194] Registered new tserver with Master: d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:19.223421 16223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:36845
W20260621 07:07:19.522442 16412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:19.522706 16412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:19.522809 16412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:19.529840 16412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:19.529940 16412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:19.530048 16412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:19.539688 16412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:42441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=33923
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:19.540949 16412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:19.542168 16412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:19.555622 16417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:19.555539 16418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:19.555583 16420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:19.556806 16412 server_base.cc:1034] running on GCE node
I20260621 07:07:19.557444 16412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:19.558385 16412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:19.559607 16412 hybrid_clock.cc:648] HybridClock initialized: now 1782025639559574 us; error 59 us; skew 500 ppm
I20260621 07:07:19.559952 16412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:19 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:19.563212 16412 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:19.565055 16412 webserver.cc:460] Webserver started at http://127.1.188.194:33923/ using document root <none> and password file <none>
I20260621 07:07:19.565802 16412 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:19.565953 16412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:19.571187 16412 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:07:19.573829 16427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.575184 16412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:07:19.575389 16412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "80ef1ded48e64b07824b085ed92c1580"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "b7979d84c3bc1f2ed674245f01a72c80"
server_key_iv: "f246e7a3d711d4a9b2b3f06a64c7217f"
server_key_version: "encryptionkey@0"
I20260621 07:07:19.576483 16412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:19.588584 16412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:19.592859 16412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:19.593147 16412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:19.594163 16412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:19.595613 16412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:19.595731 16412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.595867 16412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:19.595960 16412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:19.633715 16412 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:42441
I20260621 07:07:19.633828 16539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:42441 every 8 connection(s)
I20260621 07:07:19.635310 16412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:19.640900 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16412
I20260621 07:07:19.641628 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:35227
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=37711
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:07:19 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025639, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:19.657905 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:19.637898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:56153 (local address 127.1.188.254:35683)
0621 07:07:19.638074 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:19.638079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:19.639154 (+ 1075us) server_negotiation.cc:407] Connection header received
0621 07:07:19.641034 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:19.641042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:19.641150 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:19.641267 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:19.642794 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.643958 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.645498 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.646262 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.646412 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:19.651060 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:19.651065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:19.651068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:19.651114 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:19.654227 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:19.655168 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:19.655173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:19.655177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:19.655344 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:19.655856 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:19.655861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:19.655864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:19.656247 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:19.656393 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:19.657286 (+ 893us) server_negotiation.cc:299] Negotiation successful
0621 07:07:19.657556 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:07:19.659171 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:19.638302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:56153)
0621 07:07:19.638988 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:19.639024 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:19.640757 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:19.641470 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:19.641478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:19.642079 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:19.642621 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:19.642637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.644105 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.644109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:19.645333 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:19.645345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:19.646469 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:19.646474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:19.646657 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:19.647871 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:19.647900 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:19.650859 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:19.654384 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:19.654395 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:19.654414 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:19.655030 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:19.655488 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:19.655496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:19.655502 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:19.655740 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:19.656407 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:19.656418 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:19.656856 (+ 438us) client_negotiation.cc:769] Sending connection context
0621 07:07:19.657162 (+ 306us) client_negotiation.cc:240] Negotiation successful
0621 07:07:19.657738 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260621 07:07:19.660887 16540 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:19.661221 16540 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:19.662006 16540 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
I20260621 07:07:19.663724 16223 ts_manager.cc:194] Registered new tserver with Master: 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441)
I20260621 07:07:19.665081 16223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:56153
W20260621 07:07:19.971264 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:19.971621 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:19.971721 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:19.978696 16543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:19.978793 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:19.978968 16543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:19.988569 16543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=37711
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:19.989882 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:19.991209 16543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:20.004516 16551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:20.004570 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:20.004544 16549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:20.005681 16543 server_base.cc:1034] running on GCE node
I20260621 07:07:20.006347 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:20.007387 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:20.008698 16543 hybrid_clock.cc:648] HybridClock initialized: now 1782025640008622 us; error 57 us; skew 500 ppm
I20260621 07:07:20.009069 16543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:20 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025640, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:20.012693 16543 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:20.014539 16543 webserver.cc:460] Webserver started at http://127.1.188.195:37711/ using document root <none> and password file <none>
I20260621 07:07:20.015275 16543 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:20.015457 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:20.021265 16543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:07:20.024333 16558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:20.025815 16543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:20.026027 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "979ab4123e4640139993d5c13d890bf9"
format_stamp: "Formatted at 2026-06-21 07:07:17 on dist-test-slave-hlq1"
server_key: "985b2aa2edbce0dc5535218ce5269fe4"
server_key_iv: "8d279a3090a60b76ff559f1264517dc3"
server_key_version: "encryptionkey@0"
I20260621 07:07:20.027179 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:20.047176 16543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:20.051875 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:20.052189 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:20.053279 16543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:20.054754 16543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:20.054899 16543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:20.055063 16543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:20.055168 16543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:20.092576 16543 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:35227
I20260621 07:07:20.092666 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:35227 every 8 connection(s)
I20260621 07:07:20.094141 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:20.101400 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16543
Jun 21 07:07:20 dist-test-slave-hlq1 krb5kdc[15638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025640, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:20.117635 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.097034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:56641 (local address 127.1.188.254:35683)
0621 07:07:20.097236 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.097241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.098522 (+ 1281us) server_negotiation.cc:407] Connection header received
0621 07:07:20.100558 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.100564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.100646 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.100744 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:20.102353 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.103620 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.105305 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.105889 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.106030 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.109908 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:20.109918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:20.109923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:20.109967 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:20.114113 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.114973 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.114981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.114989 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.115128 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.115685 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.115692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.115696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.115991 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:20.116168 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.117000 (+ 832us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.117238 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":68}
I20260621 07:07:20.118371 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.102419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.1.188.254:35683)
0621 07:07:20.103016 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.103024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.103360 (+ 336us) server_negotiation.cc:407] Connection header received
0621 07:07:20.103626 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.103630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.103692 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.103805 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:20.105467 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.106369 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.107892 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.108375 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.108510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.112139 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:20.112144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:20.112148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:20.112189 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:20.115528 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.116259 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.116264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.116268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.116394 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.116847 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.116856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.116860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.117160 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:20.117439 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.117769 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.117994 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
I20260621 07:07:20.119305 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.097608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:56641)
0621 07:07:20.098350 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.098390 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.100273 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.100968 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.100976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.101501 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:20.102124 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.102142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.103804 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.103808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.105143 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.105156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.106155 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.106165 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.106437 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.107178 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:20.107202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:20.109663 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:20.114300 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:20.114317 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:20.114331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:20.114834 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:20.115254 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:20.115261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:20.115265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:20.115561 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:20.116154 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:20.116162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:20.116527 (+ 365us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.116826 (+ 299us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.117430 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":292,"threads_started":1}
I20260621 07:07:20.121007 16671 heartbeater.cc:347] Connected to a master server at 127.1.188.254:35683
I20260621 07:07:20.121381 16671 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:20.122076 16671 heartbeater.cc:511] Master 127.1.188.254:35683 requested a full tablet report, sending...
I20260621 07:07:20.123642 16223 ts_manager.cc:194] Registered new tserver with Master: 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227)
I20260621 07:07:20.124495 16223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:56641
I20260621 07:07:20.134085 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:07:20.141551 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.135134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.1.188.254:35683)
0621 07:07:20.135340 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.135344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.135801 (+ 457us) server_negotiation.cc:407] Connection header received
0621 07:07:20.135989 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.135993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.136152 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.136298 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:07:20.136816 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.137586 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.139194 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.139719 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.139826 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.140392 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:07:20.140936 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:07:20.141046 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.141247 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.141322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:07:20.142594 16223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:07:20.149901 16223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:07:20.175945 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.162326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.0.0.1:49056)
0621 07:07:20.163776 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.163810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.163956 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.165397 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.165408 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.165451 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:20.166007 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.166024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.168881 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.168890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.172209 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.172231 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.174803 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.174812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.175059 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.175121 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.175235 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.175462 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":1,"thread_start_us":13039,"threads_started":1}
I20260621 07:07:20.176697 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.162338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.1.188.194:42441)
0621 07:07:20.163184 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.163197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.165127 (+ 1930us) server_negotiation.cc:407] Connection header received
0621 07:07:20.165492 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.165503 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.165783 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.165978 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:20.167083 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.169141 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.172704 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.174690 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.174841 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.175097 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.175979 (+ 882us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.176241 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":425,"threads_started":1}
I20260621 07:07:20.177067 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.162668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.1.188.193:40043)
0621 07:07:20.164627 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.164639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.164671 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:07:20.164790 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.164800 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.165108 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.166291 (+ 1183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:20.166579 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.168666 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.172500 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.174509 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.176249 (+ 1740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.176454 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.176521 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.176698 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":1618,"threads_started":1}
I20260621 07:07:20.177564 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.162326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:42441 (local address 127.0.0.1:52922)
0621 07:07:20.164994 (+ 2668us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.165014 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.165273 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.166344 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.166351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.166386 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:20.166851 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.166864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.169396 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.169404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.172470 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.172491 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.175370 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.175379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.175699 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.175748 (+ 49us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.177045 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.177176 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2546,"mutex_wait_us":172,"thread_start_us":919,"threads_started":1}
I20260621 07:07:20.180998 16343 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec81fe1133141608dc7bab936310167 (DEFAULT_TABLE table=test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:20.181053 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec81fe1133141608dc7bab936310167 (DEFAULT_TABLE table=test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:20.182293 16343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec81fe1133141608dc7bab936310167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:20.182337 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec81fe1133141608dc7bab936310167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:20.185715 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.163881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:35227 (local address 127.0.0.1:59648)
0621 07:07:20.175173 (+ 11292us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.175191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.175297 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.178040 (+ 2743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.178045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.178071 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:20.178390 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.178402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.180679 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.180686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.182801 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.182816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.184935 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.184942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.185110 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.185144 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.185254 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.185392 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11164,"thread_start_us":962,"threads_started":1}
I20260621 07:07:20.186504 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.164554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.1.188.195:35227)
0621 07:07:20.177226 (+ 12672us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.177235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.177272 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:07:20.177381 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.177389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.177645 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.178001 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:20.178909 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.180537 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.182990 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.184436 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.184565 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.184795 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.185984 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.186159 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12468,"thread_start_us":12546,"threads_started":1}
I20260621 07:07:20.192552 16606 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec81fe1133141608dc7bab936310167 (DEFAULT_TABLE table=test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:20.193964 16606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec81fe1133141608dc7bab936310167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:20.199697 16687 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Bootstrap starting.
I20260621 07:07:20.204118 16688 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Bootstrap starting.
I20260621 07:07:20.205389 16687 tablet_bootstrap.cc:654] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:20.207206 16688 tablet_bootstrap.cc:654] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:20.207279 16687 log.cc:824] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:20.208464 16688 log.cc:824] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:20.210671 16687 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: No bootstrap required, opened a new log
I20260621 07:07:20.210863 16688 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: No bootstrap required, opened a new log
I20260621 07:07:20.211007 16687 ts_tablet_manager.cc:1397] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260621 07:07:20.211151 16688 ts_tablet_manager.cc:1397] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 07:07:20.216131 16691 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Bootstrap starting.
I20260621 07:07:20.219756 16691 tablet_bootstrap.cc:654] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:20.221011 16691 log.cc:824] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:20.221052 16687 raft_consensus.cc:348] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.221109 16688 raft_consensus.cc:348] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.221472 16687 raft_consensus.cc:374] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:20.221553 16688 raft_consensus.cc:374] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:20.221621 16687 raft_consensus.cc:729] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ef1ded48e64b07824b085ed92c1580, State: Initialized, Role: FOLLOWER
I20260621 07:07:20.221781 16688 raft_consensus.cc:729] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4676d8d8f4341808a988ca9a7a5ccbd, State: Initialized, Role: FOLLOWER
I20260621 07:07:20.222385 16687 consensus_queue.cc:260] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.222404 16688 consensus_queue.cc:260] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.224439 16691 tablet_bootstrap.cc:492] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: No bootstrap required, opened a new log
I20260621 07:07:20.224794 16691 ts_tablet_manager.cc:1397] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 07:07:20.227079 16408 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:20.229905 16540 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:20.233311 16687 ts_tablet_manager.cc:1428] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260621 07:07:20.234293 16688 ts_tablet_manager.cc:1428] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Time spent starting tablet: real 0.023s user 0.015s sys 0.008s
I20260621 07:07:20.238453 16691 raft_consensus.cc:348] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.238832 16691 raft_consensus.cc:374] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:20.238940 16691 raft_consensus.cc:729] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979ab4123e4640139993d5c13d890bf9, State: Initialized, Role: FOLLOWER
I20260621 07:07:20.239562 16691 consensus_queue.cc:260] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.240571 16671 heartbeater.cc:503] Master 127.1.188.254:35683 was elected leader, sending a full tablet report...
I20260621 07:07:20.241062 16691 ts_tablet_manager.cc:1428] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260621 07:07:20.346012 16672 tablet.cc:2369] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:20.387470 16541 tablet.cc:2369] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:20.439463 16409 tablet.cc:2369] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:07:20.527667 16695 raft_consensus.cc:482] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:07:20.528043 16695 raft_consensus.cc:504] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.530077 16695 leader_election.cc:288] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441), d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:20.537614 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.530744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:37549 (local address 127.1.188.194:42441)
0621 07:07:20.530983 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.530991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.531018 (+ 27us) server_negotiation.cc:407] Connection header received
0621 07:07:20.531295 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.531303 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.531425 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.531548 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:20.532565 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.533604 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.535663 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.536563 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.536691 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.536751 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.537145 (+ 394us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.537303 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:07:20.537642 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.530469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:42441 (local address 127.1.188.195:37549)
0621 07:07:20.530830 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.530876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.531063 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.531717 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.531726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.531852 (+ 126us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:20.532253 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.532272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.533757 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.533765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.535500 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.535512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.536721 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.536727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.536957 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.537010 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.537086 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.537224 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"spinlock_wait_cycles":24832}
I20260621 07:07:20.537689 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.530741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:54757 (local address 127.1.188.193:40043)
0621 07:07:20.530983 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.530990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.531657 (+ 667us) server_negotiation.cc:407] Connection header received
0621 07:07:20.531948 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.531954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.532075 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.532187 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:20.532924 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.533825 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.535837 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.536710 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.536877 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.536960 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.537254 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.537412 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:07:20.537672 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.530527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:40043 (local address 127.1.188.195:54757)
0621 07:07:20.531547 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:20.531567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:20.531769 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:20.532243 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:20.532250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:20.532272 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:20.532743 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.532754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.533983 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.533987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.535683 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:20.535697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.536920 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.536924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:20.537080 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.537117 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:07:20.537209 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 07:07:20.537327 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":677,"threads_started":1}
I20260621 07:07:20.539009 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec81fe1133141608dc7bab936310167" candidate_uuid: "979ab4123e4640139993d5c13d890bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ef1ded48e64b07824b085ed92c1580" is_pre_election: true
I20260621 07:07:20.539011 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec81fe1133141608dc7bab936310167" candidate_uuid: "979ab4123e4640139993d5c13d890bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" is_pre_election: true
I20260621 07:07:20.539723 16495 raft_consensus.cc:2454] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979ab4123e4640139993d5c13d890bf9 in term 0.
I20260621 07:07:20.539723 16363 raft_consensus.cc:2454] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979ab4123e4640139993d5c13d890bf9 in term 0.
I20260621 07:07:20.540586 16561 leader_election.cc:302] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80ef1ded48e64b07824b085ed92c1580, 979ab4123e4640139993d5c13d890bf9, d4676d8d8f4341808a988ca9a7a5ccbd; no voters:
I20260621 07:07:20.541117 16695 raft_consensus.cc:2781] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:07:20.541296 16695 raft_consensus.cc:482] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:07:20.541430 16695 raft_consensus.cc:3037] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:20.543938 16695 raft_consensus.cc:504] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.544603 16695 leader_election.cc:288] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [CANDIDATE]: Term 1 election: Requested vote from peers 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441), d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043)
I20260621 07:07:20.545178 16363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec81fe1133141608dc7bab936310167" candidate_uuid: "979ab4123e4640139993d5c13d890bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
I20260621 07:07:20.545181 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec81fe1133141608dc7bab936310167" candidate_uuid: "979ab4123e4640139993d5c13d890bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ef1ded48e64b07824b085ed92c1580"
I20260621 07:07:20.545418 16495 raft_consensus.cc:3037] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:20.545418 16363 raft_consensus.cc:3037] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:20.548936 16363 raft_consensus.cc:2454] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979ab4123e4640139993d5c13d890bf9 in term 1.
I20260621 07:07:20.548938 16495 raft_consensus.cc:2454] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979ab4123e4640139993d5c13d890bf9 in term 1.
I20260621 07:07:20.549499 16562 leader_election.cc:302] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80ef1ded48e64b07824b085ed92c1580, 979ab4123e4640139993d5c13d890bf9; no voters:
I20260621 07:07:20.549868 16695 raft_consensus.cc:2781] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:20.551355 16695 raft_consensus.cc:686] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 979ab4123e4640139993d5c13d890bf9, State: Running, Role: LEADER
I20260621 07:07:20.551957 16695 consensus_queue.cc:237] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } }
I20260621 07:07:20.557380 16223 catalog_manager.cc:5495] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 979ab4123e4640139993d5c13d890bf9 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "979ab4123e4640139993d5c13d890bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "979ab4123e4640139993d5c13d890bf9" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 35227 } health_report { overall_health: HEALTHY } } }
I20260621 07:07:20.654992 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.647533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.1.188.195:35227)
0621 07:07:20.647755 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.647760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.648117 (+ 357us) server_negotiation.cc:407] Connection header received
0621 07:07:20.648320 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.648325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.648426 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.648564 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:07:20.648956 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.649737 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.651832 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.652513 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.652662 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.653370 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:07:20.654204 (+ 834us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:07:20.654350 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.654565 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.654653 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 07:07:20.664778 16495 raft_consensus.cc:1261] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Refusing update from remote peer 979ab4123e4640139993d5c13d890bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:20.664784 16363 raft_consensus.cc:1261] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Refusing update from remote peer 979ab4123e4640139993d5c13d890bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:20.665890 16695 consensus_queue.cc:1035] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ef1ded48e64b07824b085ed92c1580" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:20.666265 16697 consensus_queue.cc:1035] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4676d8d8f4341808a988ca9a7a5ccbd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:20.699307 16223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:20.699734 16223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:20.703771 16223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Sending DeleteTablet for 3 replicas of tablet 3ec81fe1133141608dc7bab936310167
I20260621 07:07:20.705196 16475 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec81fe1133141608dc7bab936310167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:20 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:52922
I20260621 07:07:20.705202 16343 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec81fe1133141608dc7bab936310167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:20 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:49056
I20260621 07:07:20.705973 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16260
I20260621 07:07:20.706089 16606 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec81fe1133141608dc7bab936310167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:20 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:59648
I20260621 07:07:20.716953 16711 tablet_replica.cc:330] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: stopping tablet replica
I20260621 07:07:20.717674 16711 raft_consensus.cc:2229] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:20.718367 16711 raft_consensus.cc:2258] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:20.719250 16712 tablet_replica.cc:330] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: stopping tablet replica
I20260621 07:07:20.719782 16712 raft_consensus.cc:2229] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:20.720415 16712 raft_consensus.cc:2258] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:20.721890 16711 ts_tablet_manager.cc:1905] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:20.723104 16712 ts_tablet_manager.cc:1905] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:20.729398 16713 tablet_replica.cc:330] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: stopping tablet replica
I20260621 07:07:20.729616 16711 ts_tablet_manager.cc:1918] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:20.729849 16711 log.cc:1196] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/3ec81fe1133141608dc7bab936310167
I20260621 07:07:20.730057 16713 raft_consensus.cc:2229] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:07:20.730571 16712 ts_tablet_manager.cc:1918] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:20.730607 16711 ts_tablet_manager.cc:1939] T 3ec81fe1133141608dc7bab936310167 P 80ef1ded48e64b07824b085ed92c1580: Deleting consensus metadata
I20260621 07:07:20.730826 16712 log.cc:1196] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/3ec81fe1133141608dc7bab936310167
I20260621 07:07:20.731694 16712 ts_tablet_manager.cc:1939] T 3ec81fe1133141608dc7bab936310167 P d4676d8d8f4341808a988ca9a7a5ccbd: Deleting consensus metadata
I20260621 07:07:20.732102 16713 raft_consensus.cc:2258] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:20.733672 16210 catalog_manager.cc:4841] TS 80ef1ded48e64b07824b085ed92c1580 (127.1.188.194:42441): tablet 3ec81fe1133141608dc7bab936310167 (table test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]) successfully deleted
I20260621 07:07:20.733939 16209 catalog_manager.cc:4841] TS d4676d8d8f4341808a988ca9a7a5ccbd (127.1.188.193:40043): tablet 3ec81fe1133141608dc7bab936310167 (table test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]) successfully deleted
I20260621 07:07:20.736738 16713 ts_tablet_manager.cc:1905] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:20.743291 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.706905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.1.188.193:40043)
0621 07:07:20.707179 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.707189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.721393 (+ 14204us) server_negotiation.cc:407] Connection header received
0621 07:07:20.721617 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.721623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.721728 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.721862 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:20.725763 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.726970 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.730280 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.730986 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.731157 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.736868 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:20.736891 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:20.736915 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:20.736965 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:20.740071 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.740888 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.740896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.740899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.741039 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.741487 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.741492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.741494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.741933 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:20.742281 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.742613 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.742870 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 07:07:20.744103 16403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:20.745054 16713 ts_tablet_manager.cc:1918] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:20.745296 16713 log.cc:1196] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/3ec81fe1133141608dc7bab936310167
I20260621 07:07:20.745922 16713 ts_tablet_manager.cc:1939] T 3ec81fe1133141608dc7bab936310167 P 979ab4123e4640139993d5c13d890bf9: Deleting consensus metadata
I20260621 07:07:20.748157 16208 catalog_manager.cc:4841] TS 979ab4123e4640139993d5c13d890bf9 (127.1.188.195:35227): tablet 3ec81fe1133141608dc7bab936310167 (table test-table [id=7162abc36e3e42efb5f7fc8d5155acbd]) successfully deleted
I20260621 07:07:20.843765 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16260
I20260621 07:07:20.872349 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16412
I20260621 07:07:20.890128 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:20.873308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.1.188.194:42441)
0621 07:07:20.873504 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:20.873510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:20.873530 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:07:20.873755 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:20.873760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:20.873837 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:20.873927 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:20.875961 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.876894 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.878840 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:20.879291 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:20.879463 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:20.883190 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:20.883209 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:20.883231 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:20.883279 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:20.886630 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.887580 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.887590 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.887592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.887781 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:20.888258 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:20.888263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:20.888265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:20.888745 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:20.889087 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:20.889476 (+ 389us) server_negotiation.cc:299] Negotiation successful
0621 07:07:20.889708 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:07:20.890936 16535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:20.989502 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16412
I20260621 07:07:21.010382 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16543
I20260621 07:07:21.030229 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:21.011386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.1.188.195:35227)
0621 07:07:21.011627 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:21.011632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:21.011705 (+ 73us) server_negotiation.cc:407] Connection header received
0621 07:07:21.011998 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:21.012005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:21.012085 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:21.012199 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:21.014633 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:21.015971 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:21.018103 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:21.018670 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:21.018806 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:21.022861 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:21.022875 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:21.022897 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:21.022965 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:21.026673 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:21.027786 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:21.027801 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:21.027805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:21.028053 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:21.028505 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:21.028508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:21.028511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:21.028945 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:21.029210 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:21.029622 (+ 412us) server_negotiation.cc:299] Negotiation successful
0621 07:07:21.029818 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:07:21.031062 16666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:21.172259 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16543
I20260621 07:07:21.194420 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16190
I20260621 07:07:21.196091 16253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:21.327729 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16190
I20260621 07:07:21.354228 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--webserver_interface=127.1.188.254
--webserver_port=45301
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:35683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:21.749835 16719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:21.750236 16719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:21.750419 16719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:21.759155 16719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:21.759335 16719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:21.759454 16719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:21.759578 16719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:21.759660 16719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:21.771289 16719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:35683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=45301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:21.773522 16719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:21.775246 16719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:21.789453 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:21.789515 16727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:21.789410 16725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:21.790413 16719 server_base.cc:1034] running on GCE node
I20260621 07:07:21.791368 16719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:21.793241 16719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:21.794461 16719 hybrid_clock.cc:648] HybridClock initialized: now 1782025641794393 us; error 57 us; skew 500 ppm
I20260621 07:07:21.794891 16719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:21 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025641, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:21.798646 16719 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:21.800737 16719 webserver.cc:460] Webserver started at http://127.1.188.254:45301/ using document root <none> and password file <none>
I20260621 07:07:21.801654 16719 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:21.801817 16719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:21.808666 16719 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 07:07:21.812163 16734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:21.813807 16719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:07:21.814011 16719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "47986044021f40aa99e9dddcfee75b56"
format_stamp: "Formatted at 2026-06-21 07:07:15 on dist-test-slave-hlq1"
server_key: "d7825af3cb7b2d1ffcf89c9a7107e5ad"
server_key_iv: "6d96ef676bc0f31b18a358b7ba567680"
server_key_version: "encryptionkey@0"
I20260621 07:07:21.815166 16719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:21.834349 16719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:21.839177 16719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:21.839578 16719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:21.861099 16719 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:35683
I20260621 07:07:21.861121 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:35683 every 8 connection(s)
I20260621 07:07:21.863065 16719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:21.864241 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16719
I20260621 07:07:21.865381 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:40043
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=45707
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:07:21.874658 16786 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:07:21.880070 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap starting.
I20260621 07:07:21.904492 16786 log.cc:824] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:21.922756 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:07:21.923457 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56: Bootstrap complete.
I20260621 07:07:21.933844 16786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:21.934983 16786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Initialized, Role: FOLLOWER
I20260621 07:07:21.935778 16786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:21.936045 16786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:21.936167 16786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:21.936373 16786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:07:21.940871 16786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:21.941507 16786 leader_election.cc:302] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47986044021f40aa99e9dddcfee75b56; no voters:
I20260621 07:07:21.942298 16786 leader_election.cc:288] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:07:21.942469 16790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:07:21.943704 16790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [term 3 LEADER]: Becoming Leader. State: Replica: 47986044021f40aa99e9dddcfee75b56, State: Running, Role: LEADER
I20260621 07:07:21.944437 16790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } }
I20260621 07:07:21.944877 16786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:21.947500 16792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47986044021f40aa99e9dddcfee75b56. Latest consensus state: current_term: 3 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:21.948395 16792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:21.949458 16791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "47986044021f40aa99e9dddcfee75b56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47986044021f40aa99e9dddcfee75b56" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 35683 } } }
I20260621 07:07:21.949833 16791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47986044021f40aa99e9dddcfee75b56 [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:21.961295 16798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:21.972662 16798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:21.974709 16798 catalog_manager.cc:1259] Loaded cluster ID: 44685f87f0634f1d9afb67e06713e97c
I20260621 07:07:21.974910 16798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 07:07:21.977320 16798 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7d529f79ad google::LogMessage::Fail() at ??:0
@ 0x7f7d529fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7d529f74de google::LogMessage::Flush() at ??:0
@ 0x7f7d529f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7d6842a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7d68427976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7d684af610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7d684af439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7d55643345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7d5564691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7d55646799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7d55621eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7d588fb6db start_thread at ??:0
@ 0x7f7d5189161f clone at ??:0
W20260621 07:07:22.324769 16788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:22.325142 16788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:22.325323 16788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:22.333484 16788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:22.333652 16788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:22.333853 16788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:07:22.344440 16788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:40043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=45707
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:22.346124 16788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:22.347851 16788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:22.363232 16827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:22.363106 16828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:22.363063 16830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:22.364276 16788 server_base.cc:1034] running on GCE node
I20260621 07:07:22.364965 16788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:22.366081 16788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:22.367326 16788 hybrid_clock.cc:648] HybridClock initialized: now 1782025642367246 us; error 74 us; skew 500 ppm
I20260621 07:07:22.367717 16788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:22 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025642, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:22.374600 16788 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:07:22.376435 16788 webserver.cc:460] Webserver started at http://127.1.188.193:45707/ using document root <none> and password file <none>
I20260621 07:07:22.377287 16788 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:22.377441 16788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:22.383823 16788 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 07:07:22.386878 16837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.388499 16788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:07:22.388710 16788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "d4676d8d8f4341808a988ca9a7a5ccbd"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "9ac25d6252583d6392cf3b70f79b1bf4"
server_key_iv: "5a8bc5e14d70a10c52160d5a563faa33"
server_key_version: "encryptionkey@0"
I20260621 07:07:22.389755 16788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:22.407390 16788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:22.412101 16788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:22.412351 16788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:22.413384 16788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:22.414783 16788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:22.414886 16788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.414973 16788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:22.415035 16788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.452994 16788 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:40043
I20260621 07:07:22.453097 16949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:40043 every 8 connection(s)
I20260621 07:07:22.454758 16788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:22.460434 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.458239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:42007)
0621 07:07:22.459058 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.459332 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":310,"threads_started":1}
I20260621 07:07:22.460474 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16788
I20260621 07:07:22.461217 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:42441
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=33923
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:22.461319 16950 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:35683 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:35683: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
I20260621 07:07:22.462309 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.461927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:49317)
0621 07:07:22.462103 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.462185 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 07:07:22.463253 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.462903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:35737)
0621 07:07:22.463055 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.463136 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:07:22.463537 16950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:07:22.844460 16953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:22.844826 16953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:22.844934 16953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:22.852674 16953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:22.852795 16953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:22.852926 16953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:22.863301 16953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:42441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=33923
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:22.864933 16953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:22.866894 16953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:22.881906 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:22.881983 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:22.882019 16961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:22.883284 16953 server_base.cc:1034] running on GCE node
I20260621 07:07:22.883986 16953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:22.885025 16953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:22.886281 16953 hybrid_clock.cc:648] HybridClock initialized: now 1782025642886237 us; error 34 us; skew 500 ppm
I20260621 07:07:22.886665 16953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:22 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025642, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:22.890232 16953 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:22.891886 16953 webserver.cc:460] Webserver started at http://127.1.188.194:33923/ using document root <none> and password file <none>
I20260621 07:07:22.892697 16953 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:22.892848 16953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:22.899034 16953 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:07:22.902282 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.904160 16953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:07:22.904354 16953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "80ef1ded48e64b07824b085ed92c1580"
format_stamp: "Formatted at 2026-06-21 07:07:16 on dist-test-slave-hlq1"
server_key: "b7979d84c3bc1f2ed674245f01a72c80"
server_key_iv: "f246e7a3d711d4a9b2b3f06a64c7217f"
server_key_version: "encryptionkey@0"
I20260621 07:07:22.905521 16953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:22.940876 16953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:22.945842 16953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:22.946189 16953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:22.947424 16953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:22.948982 16953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:22.949072 16953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.949189 16953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:22.949296 16953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:22.986052 16953 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:42441
I20260621 07:07:22.986106 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:42441 every 8 connection(s)
I20260621 07:07:22.987948 16953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:22.990341 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 16953
I20260621 07:07:22.991264 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:35227
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:35683
--webserver_port=37711
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:07:22.994495 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.991917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:39227)
0621 07:07:22.992922 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.993245 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":383,"threads_started":1}
W20260621 07:07:22.995467 17081 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:35683 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:35683: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
I20260621 07:07:22.997368 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.996189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:57235)
0621 07:07:22.997163 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.997246 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":857}
I20260621 07:07:22.998376 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:22.998013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:48947)
0621 07:07:22.998169 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:22.998247 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 07:07:22.998679 17081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:07:23.377990 17084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:23.378368 17084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:23.378539 17084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:23.387264 17084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:23.387503 17084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:23.387720 17084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:23.398207 17084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=37711
--tserver_master_addrs=127.1.188.254:35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:23.399714 17084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:23.401196 17084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:23.416078 17089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:23.416046 17092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:23.416010 17090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:23.417188 17084 server_base.cc:1034] running on GCE node
I20260621 07:07:23.417794 17084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:23.418884 17084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:23.420105 17084 hybrid_clock.cc:648] HybridClock initialized: now 1782025643420047 us; error 53 us; skew 500 ppm
I20260621 07:07:23.420449 17084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:23 dist-test-slave-hlq1 krb5kdc[15638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025643, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:23.423738 17084 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:23.425197 17084 webserver.cc:460] Webserver started at http://127.1.188.195:37711/ using document root <none> and password file <none>
I20260621 07:07:23.425998 17084 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:23.426139 17084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:23.431757 17084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:07:23.434607 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:23.436070 17084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:23.436239 17084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "979ab4123e4640139993d5c13d890bf9"
format_stamp: "Formatted at 2026-06-21 07:07:17 on dist-test-slave-hlq1"
server_key: "985b2aa2edbce0dc5535218ce5269fe4"
server_key_iv: "8d279a3090a60b76ff559f1264517dc3"
server_key_version: "encryptionkey@0"
I20260621 07:07:23.437252 17084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:23.456518 17084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:23.460712 17084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:23.460999 17084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:23.462016 17084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:23.463472 17084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:23.463588 17084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:23.463723 17084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:23.463817 17084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:23.465221 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:23.464319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:59959)
0621 07:07:23.464825 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:23.464952 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":225,"threads_started":1}
I20260621 07:07:23.498641 17084 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:35227
I20260621 07:07:23.498667 17212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:35227 every 8 connection(s)
I20260621 07:07:23.500320 17084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:23.502097 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17084
I20260621 07:07:23.505843 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:23.503628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:55683)
0621 07:07:23.504637 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:23.504925 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":504,"threads_started":1}
W20260621 07:07:23.506631 17213 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:35683 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:35683: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
I20260621 07:07:23.507537 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:23.507154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:52133)
0621 07:07:23.507310 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:23.507383 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:07:23.508530 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:23.508074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:38545)
0621 07:07:23.508211 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:23.508403 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:07:23.508813 17213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:07:24.000367 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:23.999376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:37603)
0621 07:07:23.999971 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:24.000073 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":253,"threads_started":1}
I20260621 07:07:24.467151 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:24.466300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:48173)
0621 07:07:24.466838 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:24.466941 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
I20260621 07:07:24.510412 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:24.509422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:60239)
0621 07:07:24.509924 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:24.510137 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260621 07:07:25.002391 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:25.001521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:55583)
0621 07:07:25.002065 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:25.002178 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":270,"threads_started":1}
I20260621 07:07:25.469169 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:25.468322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:49221)
0621 07:07:25.468849 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:25.468947 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
I20260621 07:07:25.512295 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:25.511477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:51753)
0621 07:07:25.511992 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:25.512084 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
I20260621 07:07:26.004556 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:26.003669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:36139)
0621 07:07:26.004230 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:26.004331 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":283,"threads_started":1}
I20260621 07:07:26.471297 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:26.470337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:45685)
0621 07:07:26.470973 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:26.471067 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":340,"threads_started":1}
I20260621 07:07:26.514133 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:26.513318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:53911)
0621 07:07:26.513808 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:26.513918 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260621 07:07:27.006700 17226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:27.005657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:56167)
0621 07:07:27.006292 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:27.006398 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":322,"threads_started":1}
I20260621 07:07:27.473349 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:27.472480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:60393)
0621 07:07:27.473035 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:27.473137 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260621 07:07:27.516269 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:27.515262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:35223)
0621 07:07:27.515872 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:27.515978 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
I20260621 07:07:28.008862 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:28.008064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:57223)
0621 07:07:28.008564 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:28.008669 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":257,"threads_started":1}
I20260621 07:07:28.475348 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:28.474516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:56261)
0621 07:07:28.475035 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:28.475158 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
I20260621 07:07:28.518435 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:28.517482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:46583)
0621 07:07:28.518095 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:28.518203 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":310,"threads_started":1}
I20260621 07:07:29.010993 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:29.010040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:54991)
0621 07:07:29.010578 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:29.010694 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":271,"threads_started":1}
I20260621 07:07:29.477723 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:29.476693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:45253)
0621 07:07:29.477350 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:29.477462 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
I20260621 07:07:29.520548 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:29.519605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:36747)
0621 07:07:29.520220 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:29.520331 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":354,"threads_started":1}
I20260621 07:07:30.013415 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:30.012361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:40063)
0621 07:07:30.013010 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:30.013145 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":277,"threads_started":1}
I20260621 07:07:30.479774 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:30.478810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:48047)
0621 07:07:30.479372 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:30.479545 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":302,"threads_started":1}
I20260621 07:07:30.522985 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:30.521861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:42485)
0621 07:07:30.522513 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:30.522620 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":278,"threads_started":1}
I20260621 07:07:31.015710 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:31.014770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:44109)
0621 07:07:31.015319 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:31.015479 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260621 07:07:31.482012 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:31.481009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:33805)
0621 07:07:31.481616 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:31.481726 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":284,"threads_started":1}
I20260621 07:07:31.525141 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:31.524183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:43169)
0621 07:07:31.524808 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:31.524921 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":374,"threads_started":1}
I20260621 07:07:32.017736 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:32.016810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:34977)
0621 07:07:32.017377 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:32.017509 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":288,"threads_started":1}
I20260621 07:07:32.484299 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:32.483280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:59733)
0621 07:07:32.483966 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:32.484082 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":384,"threads_started":1}
I20260621 07:07:32.527060 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:32.526157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:36011)
0621 07:07:32.526689 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:32.526792 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":249,"threads_started":1}
I20260621 07:07:33.019896 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:33.018904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:52375)
0621 07:07:33.019564 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:33.019685 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":325,"threads_started":1}
I20260621 07:07:33.486492 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:33.485502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:49647)
0621 07:07:33.486153 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:33.486257 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":353,"threads_started":1}
I20260621 07:07:33.529081 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:33.528234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:60193)
0621 07:07:33.528759 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:33.528863 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":296,"threads_started":1}
I20260621 07:07:34.021706 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:34.020882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:36259)
0621 07:07:34.021381 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:34.021487 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260621 07:07:34.488654 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:34.487824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:59833)
0621 07:07:34.488346 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:34.488450 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":267,"threads_started":1}
I20260621 07:07:34.531001 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:34.530179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:34217)
0621 07:07:34.530673 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:34.530779 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":241,"threads_started":1}
I20260621 07:07:35.023901 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:35.022929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:57927)
0621 07:07:35.023587 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:35.023698 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":295,"threads_started":1}
I20260621 07:07:35.490764 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:35.489817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:59779)
0621 07:07:35.490393 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:35.490511 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":280,"threads_started":1}
I20260621 07:07:35.532989 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:35.532098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:38009)
0621 07:07:35.532617 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:35.532711 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260621 07:07:36.026131 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:36.025087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:53293)
0621 07:07:36.025769 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:36.025892 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":346,"threads_started":1}
I20260621 07:07:36.492864 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:36.492009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:44637)
0621 07:07:36.492533 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:36.492625 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":275,"threads_started":1}
I20260621 07:07:36.535058 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:36.534138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:36881)
0621 07:07:36.534711 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:36.534818 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":298,"threads_started":1}
I20260621 07:07:37.027992 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:37.027110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:36563)
0621 07:07:37.027663 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:37.027761 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
I20260621 07:07:37.495770 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:37.494668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:41045)
0621 07:07:37.495307 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:37.495516 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":326,"threads_started":1}
I20260621 07:07:37.536939 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:37.536103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:44079)
0621 07:07:37.536617 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:37.536713 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260621 07:07:38.030242 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.029291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.194:55971)
0621 07:07:38.029892 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:38.029998 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":343,"threads_started":1}
I20260621 07:07:38.497799 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.496877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.193:55317)
0621 07:07:38.497487 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:38.497593 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":296,"threads_started":1}
I20260621 07:07:38.515064 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16788
I20260621 07:07:38.536769 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.516623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.1.188.193:40043)
0621 07:07:38.517363 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:38.517383 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:38.517445 (+ 62us) server_negotiation.cc:407] Connection header received
0621 07:07:38.519381 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:38.519468 (+ 87us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:38.520030 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:38.520379 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:38.522096 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.523591 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.525096 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.525868 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.526035 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:38.529295 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:38.529306 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:38.529326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:38.529376 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:38.533119 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.533896 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.533919 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.533926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.534102 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.534504 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.534507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.534509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.535128 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:38.535492 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:38.535935 (+ 443us) server_negotiation.cc:299] Negotiation successful
0621 07:07:38.536420 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":327,"threads_started":1}
I20260621 07:07:38.538823 16945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:38.539232 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.538334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:35683 (local address 127.1.188.195:44439)
0621 07:07:38.538861 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:38.539005 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":287,"threads_started":1}
I20260621 07:07:38.632174 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16788
I20260621 07:07:38.652638 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 16953
I20260621 07:07:38.673362 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.653907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.1.188.194:42441)
0621 07:07:38.654570 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:38.654587 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:38.654635 (+ 48us) server_negotiation.cc:407] Connection header received
0621 07:07:38.656607 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:38.656625 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:38.657182 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:38.657603 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:38.659777 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.661147 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.662415 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.663155 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.663305 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:38.666223 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:38.666231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:38.666248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:38.666290 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:38.669815 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.670492 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.670499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.670506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.670681 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.671003 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.671007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.671008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.671894 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:38.672173 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:38.672557 (+ 384us) server_negotiation.cc:299] Negotiation successful
0621 07:07:38.673031 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":292,"threads_started":1}
I20260621 07:07:38.675031 17076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:38.770007 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 16953
I20260621 07:07:38.789074 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17084
I20260621 07:07:38.807644 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:38.790242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.1.188.195:35227)
0621 07:07:38.790879 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:38.790894 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:38.790942 (+ 48us) server_negotiation.cc:407] Connection header received
0621 07:07:38.792862 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:38.792873 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:38.793370 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:38.793682 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:38.795178 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.796602 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.797800 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:38.798463 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:38.798591 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:38.801344 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:38.801351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:38.801367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:38.801403 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:38.804428 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.805178 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.805185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.805191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.805361 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:38.805687 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:38.805690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:38.805692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:38.806300 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:38.806552 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:38.806863 (+ 311us) server_negotiation.cc:299] Negotiation successful
0621 07:07:38.807247 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":303,"threads_started":1}
I20260621 07:07:38.809182 17208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:38.894829 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17084
2026-06-21T07:07:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25058 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:07:38 dist-test-slave-hlq1 krb5kdc[17270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:07:38 dist-test-slave-hlq1 krb5kdc[17270](info): set up 2 sockets
Jun 21 07:07:38 dist-test-slave-hlq1 krb5kdc[17270](info): commencing operation
krb5kdc: starting...
W20260621 07:07:40.004328 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:07:40 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:07:40.248992 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:40.605202 17283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:40.605564 17283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:40.605728 17283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:40.613005 17283 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:07:40.613142 17283 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:07:40.613235 17283 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:07:40.613332 17283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:07:40.613432 17283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:40.613554 17283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:40.613643 17283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:07:40.613729 17283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:07:40.623746 17283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:40.625043 17283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:40.626377 17283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:40.638432 17289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:40.638502 17288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:40.638497 17291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:40.639744 17283 server_base.cc:1034] running on GCE node
I20260621 07:07:40.640628 17283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:40.642068 17283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:40.643352 17283 hybrid_clock.cc:648] HybridClock initialized: now 1782025660643279 us; error 45 us; skew 500 ppm
I20260621 07:07:40.643774 17283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:40 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:40.646934 17283 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:07:40.648429 17283 webserver.cc:460] Webserver started at http://127.1.188.254:33827/ using document root <none> and password file <none>
I20260621 07:07:40.649091 17283 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:40.649184 17283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:40.649436 17283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:40.652442 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "d1389a0f7a7449a681f56fc7ff5137cc"
format_stamp: "Formatted at 2026-06-21 07:07:40 on dist-test-slave-hlq1"
server_key: "aee35f6ba1fb8550ebb982f3b94d31cd"
server_key_iv: "e16af1d833147d1b43b70a7e3534d124"
server_key_version: "encryptionkey@0"
I20260621 07:07:40.653191 17283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "d1389a0f7a7449a681f56fc7ff5137cc"
format_stamp: "Formatted at 2026-06-21 07:07:40 on dist-test-slave-hlq1"
server_key: "aee35f6ba1fb8550ebb982f3b94d31cd"
server_key_iv: "e16af1d833147d1b43b70a7e3534d124"
server_key_version: "encryptionkey@0"
I20260621 07:07:40.658423 17283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:07:40.661985 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:40.663556 17283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:07:40.663712 17283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "d1389a0f7a7449a681f56fc7ff5137cc"
format_stamp: "Formatted at 2026-06-21 07:07:40 on dist-test-slave-hlq1"
server_key: "aee35f6ba1fb8550ebb982f3b94d31cd"
server_key_iv: "e16af1d833147d1b43b70a7e3534d124"
server_key_version: "encryptionkey@0"
I20260621 07:07:40.663915 17283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:40.682156 17283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:40.686235 17283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:40.686468 17283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:40.706086 17283 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42059
I20260621 07:07:40.706099 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42059 every 8 connection(s)
I20260621 07:07:40.707559 17283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:07:40.711573 17350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:40.711927 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17283
I20260621 07:07:40.712198 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:07:40.712875 1779 external_mini_cluster.cc:1428] Setting key 84c975418bd1af7ac193a8d993671be7
I20260621 07:07:40.726254 17350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap starting.
Jun 21 07:07:40 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:40.729730 17350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:40.731072 17350 log.cc:824] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:40.735272 17350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: No bootstrap required, opened a new log
I20260621 07:07:40.740109 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:40.714946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.1.188.254:42059)
0621 07:07:40.715870 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:40.715884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:40.715932 (+ 48us) server_negotiation.cc:407] Connection header received
0621 07:07:40.718405 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:40.718434 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:40.718938 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:40.719427 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:40.720940 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:40.722978 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:40.724480 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:40.725689 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:40.725882 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:40.731027 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:40.731039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:40.731064 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:40.731131 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:40.735242 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:40.736141 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:40.736155 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:40.736166 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:40.736376 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:40.736762 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:40.736767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:40.736770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:40.737512 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:40.737837 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:40.738173 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 07:07:40.738667 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":414,"threads_started":1}
I20260621 07:07:40.745204 17350 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:07:40.745560 17350 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:40.745638 17350 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Initialized, Role: FOLLOWER
I20260621 07:07:40.746145 17350 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:07:40.746351 17350 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:07:40.746500 17350 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:07:40.746673 17350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:40.749387 17350 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:07:40.749958 17350 leader_election.cc:302] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1389a0f7a7449a681f56fc7ff5137cc; no voters:
I20260621 07:07:40.750553 17350 leader_election.cc:288] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:07:40.750710 17355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:40.751981 17355 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 LEADER]: Becoming Leader. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Running, Role: LEADER
I20260621 07:07:40.752583 17355 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:07:40.752869 17350 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:07:40.755942 17356 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:07:40.756453 17356 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:40.757005 17357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1389a0f7a7449a681f56fc7ff5137cc. Latest consensus state: current_term: 1 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:07:40.757375 17357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:07:40.758199 17364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:07:40.764586 17364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:07:40.775345 17364 catalog_manager.cc:1347] Generated new cluster ID: 6a7ffb42d24d488487c4d4e2099c6b33
I20260621 07:07:40.775529 17364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:07:40.793419 17364 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:07:40.794816 17364 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:07:40.802225 17364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Generated new TSK 0
I20260621 07:07:40.803066 17364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:07:40.858533 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:41.190297 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:41.190572 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:41.190666 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:41.197770 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:41.197888 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:41.198001 17376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:07:41.207698 17376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:41.208954 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:41.210366 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:41.223501 17381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:41.223501 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:41.223562 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:41.224434 17376 server_base.cc:1034] running on GCE node
I20260621 07:07:41.225073 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:41.226078 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:41.227308 17376 hybrid_clock.cc:648] HybridClock initialized: now 1782025661227248 us; error 52 us; skew 500 ppm
I20260621 07:07:41.227694 17376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:41 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025661, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:41.230973 17376 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:07:41.232542 17376 webserver.cc:460] Webserver started at http://127.1.188.193:36363/ using document root <none> and password file <none>
I20260621 07:07:41.233230 17376 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:41.233366 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:41.233704 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:41.236769 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "35bf04323879438dbb46f95969b638dd"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "73a8265f20dfdaf6ea999b8647cba5fc"
server_key_iv: "bf628e73d3a546a56276d623a23d0a18"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.237550 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "35bf04323879438dbb46f95969b638dd"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "73a8265f20dfdaf6ea999b8647cba5fc"
server_key_iv: "bf628e73d3a546a56276d623a23d0a18"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.242838 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:07:41.246404 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.247809 17376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:41.247980 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "35bf04323879438dbb46f95969b638dd"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "73a8265f20dfdaf6ea999b8647cba5fc"
server_key_iv: "bf628e73d3a546a56276d623a23d0a18"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.248255 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:41.274834 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:41.279495 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:41.279804 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:41.280793 17376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:41.282235 17376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:41.282339 17376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.282493 17376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:41.282570 17376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.317356 17376 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33235
I20260621 07:07:41.317445 17503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33235 every 8 connection(s)
I20260621 07:07:41.318681 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:07:41.322919 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17376
I20260621 07:07:41.323167 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:07:41.323910 1779 external_mini_cluster.cc:1428] Setting key 59820c750af5f0dcc0b3b1ac6de18fd6
Jun 21 07:07:41 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025661, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:41.340221 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:41.321497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:53121 (local address 127.1.188.254:42059)
0621 07:07:41.322001 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:41.322006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:41.323034 (+ 1028us) server_negotiation.cc:407] Connection header received
0621 07:07:41.325087 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:41.325092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:41.325163 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:41.325304 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:41.326845 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.328101 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.329632 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.330367 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.330515 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:41.334197 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:41.334202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:41.334206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:41.334237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:41.336991 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:41.337902 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:41.337906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:41.337908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:41.338023 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:41.338472 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:41.338475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:41.338477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:41.338726 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:41.338842 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:41.339785 (+ 943us) server_negotiation.cc:299] Negotiation successful
0621 07:07:41.339954 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
I20260621 07:07:41.341761 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:41.322029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:53121)
0621 07:07:41.322847 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:41.322898 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:41.324854 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:41.325468 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:41.325475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:41.326135 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:41.326679 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:41.326696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.328255 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.328259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:41.329484 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:41.329496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.330574 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.330578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:41.330774 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:41.331526 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:41.331550 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:41.333960 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:41.337128 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:41.337139 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:41.337157 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:41.337786 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:41.338124 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:41.338130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:41.338133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:41.338367 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:41.338868 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:41.338879 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:41.339321 (+ 442us) client_negotiation.cc:769] Sending connection context
0621 07:07:41.339666 (+ 345us) client_negotiation.cc:240] Negotiation successful
0621 07:07:41.340252 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":346,"threads_started":1}
I20260621 07:07:41.343320 17504 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:07:41.343653 17504 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:41.344411 17504 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
I20260621 07:07:41.346535 17315 ts_manager.cc:194] Registered new tserver with Master: 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235)
I20260621 07:07:41.348059 17315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:53121
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:07:41.378172 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:41.710562 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:41.710873 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:41.711043 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:41.719509 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:41.719659 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:41.719842 17510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:07:41.730118 17510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:41.731539 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:41.733048 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:41.747301 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:41.747315 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:41.747573 17518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:41.749006 17510 server_base.cc:1034] running on GCE node
I20260621 07:07:41.749632 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:41.750675 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:41.751906 17510 hybrid_clock.cc:648] HybridClock initialized: now 1782025661751849 us; error 51 us; skew 500 ppm
I20260621 07:07:41.752228 17510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:41 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025661, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:41.755523 17510 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:07:41.756861 17510 webserver.cc:460] Webserver started at http://127.1.188.194:40485/ using document root <none> and password file <none>
I20260621 07:07:41.757525 17510 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:41.757622 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:41.757948 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:41.760932 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "c1e30f69ea5f485ba878b393fe1792c0"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "d2ef70134e22ac4627f3d007c3131d98"
server_key_iv: "9d603e5e657157bcb77cb4fe7ba608c0"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.761622 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "c1e30f69ea5f485ba878b393fe1792c0"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "d2ef70134e22ac4627f3d007c3131d98"
server_key_iv: "9d603e5e657157bcb77cb4fe7ba608c0"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.766654 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 07:07:41.770223 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.771672 17510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:07:41.771843 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "c1e30f69ea5f485ba878b393fe1792c0"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "d2ef70134e22ac4627f3d007c3131d98"
server_key_iv: "9d603e5e657157bcb77cb4fe7ba608c0"
server_key_version: "encryptionkey@0"
I20260621 07:07:41.772035 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:41.794585 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:41.799181 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:41.799535 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:41.800530 17510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:41.801904 17510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:41.802013 17510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.802165 17510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:41.802237 17510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:41.837092 17510 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:44025
I20260621 07:07:41.837141 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:44025 every 8 connection(s)
I20260621 07:07:41.838409 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:07:41.841226 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17510
I20260621 07:07:41.841449 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:07:41.841972 1779 external_mini_cluster.cc:1428] Setting key f8c55a396408866c0dd9fa2de93937b2
Jun 21 07:07:41 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025661, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:41.860466 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:41.841005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:53517 (local address 127.1.188.254:42059)
0621 07:07:41.841564 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:41.841570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:41.842638 (+ 1068us) server_negotiation.cc:407] Connection header received
0621 07:07:41.844656 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:41.844663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:41.844754 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:41.844868 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:41.846508 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.847685 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.849275 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.850023 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.850167 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:41.854565 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:41.854570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:41.854574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:41.854605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:41.857257 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:41.857974 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:41.857981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:41.857984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:41.858159 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:41.858591 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:41.858599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:41.858604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:41.858886 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:41.859040 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:41.859855 (+ 815us) server_negotiation.cc:299] Negotiation successful
0621 07:07:41.860079 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":274,"threads_started":1}
I20260621 07:07:41.861507 17641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:41.841575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:53517)
0621 07:07:41.842453 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:41.842506 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:41.844390 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:41.845172 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:41.845180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:41.845708 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:41.846330 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:41.846348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.847852 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.847855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:41.849121 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:41.849138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:41.850299 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:41.850304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:41.850641 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:41.851464 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:41.851503 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:41.854331 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:41.857396 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:41.857408 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:41.857427 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:41.857852 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:41.858271 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:41.858276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:41.858282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:41.858481 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:41.859010 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:41.859018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:41.859468 (+ 450us) client_negotiation.cc:769] Sending connection context
0621 07:07:41.859751 (+ 283us) client_negotiation.cc:240] Negotiation successful
0621 07:07:41.860284 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":351,"threads_started":1}
I20260621 07:07:41.862967 17638 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:07:41.863353 17638 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:41.864284 17638 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:07:41.865994 17315 ts_manager.cc:194] Registered new tserver with Master: c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025)
I20260621 07:07:41.866972 17315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:53517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:07:41.895279 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:07:42.218211 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:07:42.218503 17644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:07:42.218694 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:07:42.226121 17644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:07:42.226303 17644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:07:42.226493 17644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:07:42.236683 17644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:07:42.238016 17644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:07:42.239317 17644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:07:42.252996 17649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:42.252971 17652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:07:42.253202 17650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:07:42.253935 17644 server_base.cc:1034] running on GCE node
I20260621 07:07:42.254592 17644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:07:42.255628 17644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:07:42.256877 17644 hybrid_clock.cc:648] HybridClock initialized: now 1782025662256839 us; error 52 us; skew 500 ppm
I20260621 07:07:42.257205 17644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:07:42 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025662, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:07:42.260783 17644 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:07:42.262212 17644 webserver.cc:460] Webserver started at http://127.1.188.195:39093/ using document root <none> and password file <none>
I20260621 07:07:42.262903 17644 fs_manager.cc:359] Metadata directory not provided
I20260621 07:07:42.263051 17644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:07:42.263370 17644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:07:42.266377 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "c64bc44dcf9a49ae9dc280b8ea641197"
format_stamp: "Formatted at 2026-06-21 07:07:42 on dist-test-slave-hlq1"
server_key: "e7beff4c9fafce2a1142a7cbbafc9a7c"
server_key_iv: "253e2223525e215d9757ea7b4dd6ed41"
server_key_version: "encryptionkey@0"
I20260621 07:07:42.267154 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "c64bc44dcf9a49ae9dc280b8ea641197"
format_stamp: "Formatted at 2026-06-21 07:07:42 on dist-test-slave-hlq1"
server_key: "e7beff4c9fafce2a1142a7cbbafc9a7c"
server_key_iv: "253e2223525e215d9757ea7b4dd6ed41"
server_key_version: "encryptionkey@0"
I20260621 07:07:42.272975 17644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:07:42.276511 17659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:42.277925 17644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:07:42.278106 17644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "c64bc44dcf9a49ae9dc280b8ea641197"
format_stamp: "Formatted at 2026-06-21 07:07:42 on dist-test-slave-hlq1"
server_key: "e7beff4c9fafce2a1142a7cbbafc9a7c"
server_key_iv: "253e2223525e215d9757ea7b4dd6ed41"
server_key_version: "encryptionkey@0"
I20260621 07:07:42.278311 17644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:07:42.298221 17644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:07:42.302750 17644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:07:42.303071 17644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:07:42.304188 17644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:07:42.305564 17644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:07:42.305671 17644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:42.305823 17644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:07:42.305896 17644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:07:42.340353 17644 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:36037
I20260621 07:07:42.340410 17771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:36037 every 8 connection(s)
I20260621 07:07:42.341626 17644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:07:42.347772 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17644
I20260621 07:07:42.347970 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:07:42.348434 1779 external_mini_cluster.cc:1428] Setting key cd94d566b585e4003b688de190d6b056
I20260621 07:07:42.351261 17504 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
Jun 21 07:07:42 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025662, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:07:42.362648 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.344302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:37801 (local address 127.1.188.254:42059)
0621 07:07:42.344530 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.344534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.345686 (+ 1152us) server_negotiation.cc:407] Connection header received
0621 07:07:42.347567 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.347573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.347683 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.347794 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:42.349330 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.350105 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.352160 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.352711 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.352869 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.356631 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:42.356637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:42.356640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:42.356672 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:42.359581 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:42.360290 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:42.360294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:42.360296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:42.360413 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:42.360827 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:42.360832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:42.360834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:42.361169 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:42.361296 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.362073 (+ 777us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.362318 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260621 07:07:42.363874 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.344770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:37801)
0621 07:07:42.345510 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.345548 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.347263 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.348019 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.348027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.348631 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:07:42.349167 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.349183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.350263 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.350268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.351999 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.352051 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.352931 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.352940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.353271 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.353983 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:07:42.354010 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:07:42.356358 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:07:42.359712 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:42.359719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:42.359731 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:42.360180 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:42.360501 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:07:42.360504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:07:42.360507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:07:42.360698 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:07:42.361309 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:07:42.361314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:07:42.361622 (+ 308us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.361953 (+ 331us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.362459 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":307,"threads_started":1}
I20260621 07:07:42.365216 17772 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:07:42.365476 17772 heartbeater.cc:464] Registering TS with master...
I20260621 07:07:42.366122 17772 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
I20260621 07:07:42.367499 17315 ts_manager.cc:194] Registered new tserver with Master: c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195:36037)
I20260621 07:07:42.368532 17315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:37801
I20260621 07:07:42.376201 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:07:42.394235 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.380889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.1.188.254:42059)
0621 07:07:42.381062 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.381066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.381555 (+ 489us) server_negotiation.cc:407] Connection header received
0621 07:07:42.381706 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.381709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.381782 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.381883 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:42.383743 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.384452 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.385815 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.386246 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.386338 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.388927 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:42.388932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:42.388935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:42.388966 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:42.391660 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:42.392363 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:42.392367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:42.392369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:42.392498 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:42.392911 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:42.392915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:42.392917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:42.393267 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:42.393468 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.393798 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.393966 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:07:42.398716 17315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:07:42.405774 17315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:07:42.430156 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.418295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.1.188.193:33235)
0621 07:07:42.420975 (+ 2680us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.420985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.421499 (+ 514us) server_negotiation.cc:407] Connection header received
0621 07:07:42.421952 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.421960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.422215 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.422378 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:42.423477 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.425133 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.427953 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.428875 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.429026 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.429266 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.429616 (+ 350us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.429779 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2498,"thread_start_us":2338,"threads_started":1}
I20260621 07:07:42.430763 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.418106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33235 (local address 127.0.0.1:34490)
0621 07:07:42.421344 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.421389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.421669 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.422723 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.422730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.422772 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:42.423215 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.423234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.425304 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.425311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.427794 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.427806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.429090 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.429096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.429318 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.429388 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.430163 (+ 775us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.430362 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"mutex_wait_us":1,"thread_start_us":4536,"threads_started":1}
I20260621 07:07:42.432350 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.420652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44025 (local address 127.0.0.1:58948)
0621 07:07:42.424523 (+ 3871us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.424537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.424681 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.425506 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.425510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.425536 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:42.426027 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.426043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.427342 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.427348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.429664 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.429677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.431553 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.431559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.431741 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.431778 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.431878 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.432016 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3769,"thread_start_us":3163,"threads_started":1}
I20260621 07:07:42.432829 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.419262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.1.188.195:36037)
0621 07:07:42.422546 (+ 3284us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.422554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.424345 (+ 1791us) server_negotiation.cc:407] Connection header received
0621 07:07:42.424453 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.424460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.424676 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.424841 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:42.425788 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.427389 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.430173 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.431559 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.431686 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.431895 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.432224 (+ 329us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.432366 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2071,"thread_start_us":1929,"threads_started":1}
I20260621 07:07:42.433275 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.418587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:36037 (local address 127.0.0.1:58322)
0621 07:07:42.424090 (+ 5503us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.424108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.424227 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.425074 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.425079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.425103 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:42.425547 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.425568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.427517 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.427523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.429998 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.430011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.431848 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.431858 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.432032 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.432054 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.432178 (+ 124us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.432299 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5378,"mutex_wait_us":3,"thread_start_us":5678,"threads_started":1}
I20260621 07:07:42.433449 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.421203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.1.188.194:44025)
0621 07:07:42.423663 (+ 2460us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.423671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.424633 (+ 962us) server_negotiation.cc:407] Connection header received
0621 07:07:42.424843 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.424848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.425055 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.425237 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:42.426209 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.427206 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.429832 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.431328 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.431499 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.431685 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.432444 (+ 759us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.432618 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2273,"thread_start_us":2105,"threads_started":1}
I20260621 07:07:42.434845 17439 tablet_service.cc:1467] Processing CreateTablet for tablet 318633c9de7a4e9489743c3573c3ae54 (DEFAULT_TABLE table=test-table [id=7412df2d75a24304924920b9a5fb68df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:42.436410 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318633c9de7a4e9489743c3573c3ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:42.436388 17573 tablet_service.cc:1467] Processing CreateTablet for tablet 318633c9de7a4e9489743c3573c3ae54 (DEFAULT_TABLE table=test-table [id=7412df2d75a24304924920b9a5fb68df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:42.436764 17707 tablet_service.cc:1467] Processing CreateTablet for tablet 318633c9de7a4e9489743c3573c3ae54 (DEFAULT_TABLE table=test-table [id=7412df2d75a24304924920b9a5fb68df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:07:42.437342 17573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318633c9de7a4e9489743c3573c3ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:42.438012 17707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 318633c9de7a4e9489743c3573c3ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:07:42.448678 17790 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Bootstrap starting.
I20260621 07:07:42.450874 17791 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Bootstrap starting.
I20260621 07:07:42.452776 17792 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Bootstrap starting.
I20260621 07:07:42.453657 17790 tablet_bootstrap.cc:654] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:42.454608 17791 tablet_bootstrap.cc:654] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:42.454988 17790 log.cc:824] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:42.455745 17791 log.cc:824] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:42.456280 17792 tablet_bootstrap.cc:654] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Neither blocks nor log segments found. Creating new log.
I20260621 07:07:42.457454 17792 log.cc:824] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Log is configured to *not* fsync() on all Append() calls
I20260621 07:07:42.458321 17790 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: No bootstrap required, opened a new log
I20260621 07:07:42.458631 17790 ts_tablet_manager.cc:1397] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260621 07:07:42.459988 17791 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: No bootstrap required, opened a new log
I20260621 07:07:42.460253 17791 ts_tablet_manager.cc:1397] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 07:07:42.461385 17792 tablet_bootstrap.cc:492] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: No bootstrap required, opened a new log
I20260621 07:07:42.461637 17792 ts_tablet_manager.cc:1397] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 07:07:42.467729 17791 raft_consensus.cc:348] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.468137 17791 raft_consensus.cc:374] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:42.468241 17791 raft_consensus.cc:729] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bf04323879438dbb46f95969b638dd, State: Initialized, Role: FOLLOWER
I20260621 07:07:42.468922 17791 consensus_queue.cc:260] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.469305 17790 raft_consensus.cc:348] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.469712 17790 raft_consensus.cc:374] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:42.469807 17790 raft_consensus.cc:729] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e30f69ea5f485ba878b393fe1792c0, State: Initialized, Role: FOLLOWER
I20260621 07:07:42.470563 17791 ts_tablet_manager.cc:1428] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260621 07:07:42.470467 17790 consensus_queue.cc:260] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.471596 17638 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
I20260621 07:07:42.472272 17790 ts_tablet_manager.cc:1428] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260621 07:07:42.473014 17792 raft_consensus.cc:348] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.473438 17792 raft_consensus.cc:374] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:07:42.473555 17792 raft_consensus.cc:729] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64bc44dcf9a49ae9dc280b8ea641197, State: Initialized, Role: FOLLOWER
I20260621 07:07:42.474169 17792 consensus_queue.cc:260] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.475315 17772 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
I20260621 07:07:42.475976 17792 ts_tablet_manager.cc:1428] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260621 07:07:42.571384 17505 tablet.cc:2369] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:42.590572 17639 tablet.cc:2369] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:07:42.593261 17773 tablet.cc:2369] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:07:42.860663 17798 raft_consensus.cc:482] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:07:42.861085 17798 raft_consensus.cc:504] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.863204 17798 leader_election.cc:288] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235), c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025)
I20260621 07:07:42.870463 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.863546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33235 (local address 127.1.188.195:55665)
0621 07:07:42.863758 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.863781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.863957 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.864504 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.864509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.864542 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:42.864800 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.864816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.866144 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.866148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.868250 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.868267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.869787 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.869791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.869938 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.869967 (+ 29us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.870044 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.870155 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 07:07:42.870558 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.863772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:55665 (local address 127.1.188.193:33235)
0621 07:07:42.864071 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.864080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.864110 (+ 30us) server_negotiation.cc:407] Connection header received
0621 07:07:42.864228 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.864237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.864331 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.864454 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:42.864945 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.865993 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.868436 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.869633 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.869809 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.869900 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.870103 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.870255 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260621 07:07:42.871346 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.863657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44025 (local address 127.1.188.195:38537)
0621 07:07:42.864601 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0621 07:07:42.864620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:07:42.864749 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:07:42.865169 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:07:42.865173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:07:42.865193 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:07:42.865488 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.865495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.866724 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.866729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.868795 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:07:42.868807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.870520 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.870528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:07:42.870739 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.870775 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:07:42.870913 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 07:07:42.871047 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":587,"threads_started":1}
I20260621 07:07:42.871647 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.864021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:38537 (local address 127.1.188.194:44025)
0621 07:07:42.864246 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.864254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.864722 (+ 468us) server_negotiation.cc:407] Connection header received
0621 07:07:42.864929 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.864934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.865045 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.865148 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:07:42.865639 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.866550 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.868956 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.870280 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:42.870437 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:42.870512 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:42.870976 (+ 464us) server_negotiation.cc:299] Negotiation successful
0621 07:07:42.871276 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:07:42.872082 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318633c9de7a4e9489743c3573c3ae54" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35bf04323879438dbb46f95969b638dd" is_pre_election: true
I20260621 07:07:42.872769 17459 raft_consensus.cc:2454] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 0.
I20260621 07:07:42.872758 17593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318633c9de7a4e9489743c3573c3ae54" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e30f69ea5f485ba878b393fe1792c0" is_pre_election: true
I20260621 07:07:42.873303 17593 raft_consensus.cc:2454] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 0.
I20260621 07:07:42.873633 17661 leader_election.cc:302] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bf04323879438dbb46f95969b638dd, c64bc44dcf9a49ae9dc280b8ea641197; no voters:
I20260621 07:07:42.874163 17798 raft_consensus.cc:2781] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:07:42.874341 17798 raft_consensus.cc:482] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:07:42.874473 17798 raft_consensus.cc:3037] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:42.877471 17798 raft_consensus.cc:504] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.878348 17798 leader_election.cc:288] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 election: Requested vote from peers 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235), c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025)
I20260621 07:07:42.878881 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318633c9de7a4e9489743c3573c3ae54" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35bf04323879438dbb46f95969b638dd"
I20260621 07:07:42.878940 17593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "318633c9de7a4e9489743c3573c3ae54" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e30f69ea5f485ba878b393fe1792c0"
I20260621 07:07:42.879189 17593 raft_consensus.cc:3037] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:42.879181 17459 raft_consensus.cc:3037] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:07:42.882881 17593 raft_consensus.cc:2454] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 1.
I20260621 07:07:42.882881 17459 raft_consensus.cc:2454] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 1.
I20260621 07:07:42.883383 17660 leader_election.cc:302] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e30f69ea5f485ba878b393fe1792c0, c64bc44dcf9a49ae9dc280b8ea641197; no voters:
I20260621 07:07:42.883814 17798 raft_consensus.cc:2781] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:07:42.884388 17798 raft_consensus.cc:686] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 LEADER]: Becoming Leader. State: Replica: c64bc44dcf9a49ae9dc280b8ea641197, State: Running, Role: LEADER
I20260621 07:07:42.884949 17798 consensus_queue.cc:237] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } }
I20260621 07:07:42.890592 17315 catalog_manager.cc:5495] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 reported cstate change: term changed from 0 to 1, leader changed from <none> to c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260621 07:07:43.004347 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:42.996951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.1.188.195:36037)
0621 07:07:42.997562 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:42.997568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:42.997750 (+ 182us) server_negotiation.cc:407] Connection header received
0621 07:07:42.997990 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:42.997995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:42.998105 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:42.998271 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:07:42.998754 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:42.999645 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.001342 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.001790 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.001895 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:43.002725 (+ 830us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:07:43.003704 (+ 979us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:07:43.003880 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:43.003966 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 07:07:43.004055 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":321,"threads_started":1}
I20260621 07:07:43.014523 17593 raft_consensus.cc:1261] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Refusing update from remote peer c64bc44dcf9a49ae9dc280b8ea641197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:43.014523 17459 raft_consensus.cc:1261] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Refusing update from remote peer c64bc44dcf9a49ae9dc280b8ea641197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:07:43.015686 17800 consensus_queue.cc:1035] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:43.016058 17798 consensus_queue.cc:1035] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:07:43.028272 17805 mvcc.cc:205] Tried to move back new op lower bound from 7299177115695611904 to 7299177115181281280. Current Snapshot: MvccSnapshot[applied={T|T < 7299177115695611904}]
I20260621 07:07:43.044332 17313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:43.044736 17313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:07:43.049279 17313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Sending DeleteTablet for 3 replicas of tablet 318633c9de7a4e9489743c3573c3ae54
I20260621 07:07:43.050753 17573 tablet_service.cc:1514] Processing DeleteTablet for tablet 318633c9de7a4e9489743c3573c3ae54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:43 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58948
I20260621 07:07:43.050755 17439 tablet_service.cc:1514] Processing DeleteTablet for tablet 318633c9de7a4e9489743c3573c3ae54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:43 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:34490
I20260621 07:07:43.050959 17707 tablet_service.cc:1514] Processing DeleteTablet for tablet 318633c9de7a4e9489743c3573c3ae54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:07:43 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:58322
I20260621 07:07:43.052073 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17376
I20260621 07:07:43.069116 17817 tablet_replica.cc:330] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: stopping tablet replica
I20260621 07:07:43.069326 17819 tablet_replica.cc:330] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: stopping tablet replica
I20260621 07:07:43.069775 17817 raft_consensus.cc:2229] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:43.069775 17819 raft_consensus.cc:2229] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:07:43.070461 17817 raft_consensus.cc:2258] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:43.070566 17819 raft_consensus.cc:2258] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:43.071910 17818 tablet_replica.cc:330] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: stopping tablet replica
I20260621 07:07:43.072464 17818 raft_consensus.cc:2229] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:07:43.073019 17818 raft_consensus.cc:2258] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:07:43.073787 17819 ts_tablet_manager.cc:1905] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:43.075111 17818 ts_tablet_manager.cc:1905] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:43.077350 17817 ts_tablet_manager.cc:1905] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:07:43.081980 17819 ts_tablet_manager.cc:1918] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:43.082157 17819 log.cc:1196] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/318633c9de7a4e9489743c3573c3ae54
I20260621 07:07:43.082800 17819 ts_tablet_manager.cc:1939] T 318633c9de7a4e9489743c3573c3ae54 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting consensus metadata
I20260621 07:07:43.082996 17818 ts_tablet_manager.cc:1918] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:43.083168 17818 log.cc:1196] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/318633c9de7a4e9489743c3573c3ae54
I20260621 07:07:43.083725 17818 ts_tablet_manager.cc:1939] T 318633c9de7a4e9489743c3573c3ae54 P 35bf04323879438dbb46f95969b638dd: Deleting consensus metadata
I20260621 07:07:43.085088 17302 catalog_manager.cc:4841] TS c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195:36037): tablet 318633c9de7a4e9489743c3573c3ae54 (table test-table [id=7412df2d75a24304924920b9a5fb68df]) successfully deleted
I20260621 07:07:43.085258 17817 ts_tablet_manager.cc:1918] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:07:43.085429 17817 log.cc:1196] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/318633c9de7a4e9489743c3573c3ae54
I20260621 07:07:43.085989 17817 ts_tablet_manager.cc:1939] T 318633c9de7a4e9489743c3573c3ae54 P c1e30f69ea5f485ba878b393fe1792c0: Deleting consensus metadata
Jun 21 07:07:43 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:07:43.086531 17300 catalog_manager.cc:4841] TS 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235): tablet 318633c9de7a4e9489743c3573c3ae54 (table test-table [id=7412df2d75a24304924920b9a5fb68df]) successfully deleted
I20260621 07:07:43.087908 17299 catalog_manager.cc:4841] TS c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025): tablet 318633c9de7a4e9489743c3573c3ae54 (table test-table [id=7412df2d75a24304924920b9a5fb68df]) successfully deleted
I20260621 07:07:43.096252 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:43.053424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.1.188.193:33235)
0621 07:07:43.053710 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:43.053720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:43.067304 (+ 13584us) server_negotiation.cc:407] Connection header received
0621 07:07:43.068031 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:43.068041 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:43.068198 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:43.068368 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:43.071751 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.073043 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.075417 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.076169 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.076277 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:43.088349 (+ 12072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:43.088364 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:43.088386 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:43.088436 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:43.092442 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.093396 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.093406 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.093410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.093593 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.094022 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.094028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.094033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.094607 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:43.094960 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:43.095256 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:07:43.095589 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260621 07:07:43.097173 17499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:43.194512 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17376
I20260621 07:07:43.215216 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17510
Jun 21 07:07:43 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:07:43.233032 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:43.216242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.1.188.194:44025)
0621 07:07:43.216492 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:43.216498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:43.216521 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:07:43.216739 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:43.216744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:43.216833 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:43.216956 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:43.218538 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.219339 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.220816 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.221284 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.221436 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:43.226480 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:43.226498 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:43.226519 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:43.226563 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:43.229850 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.230608 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.230616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.230619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.230748 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.231161 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.231166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.231174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.231662 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:43.231991 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:43.232337 (+ 346us) server_negotiation.cc:299] Negotiation successful
0621 07:07:43.232625 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 07:07:43.233855 17633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:43.334151 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17510
I20260621 07:07:43.354576 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17644
Jun 21 07:07:43 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:07:43.373734 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:07:43.355630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.1.188.195:36037)
0621 07:07:43.355836 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 07:07:43.355848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:07:43.355913 (+ 65us) server_negotiation.cc:407] Connection header received
0621 07:07:43.356174 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:07:43.356179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:07:43.356255 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:07:43.356377 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:07:43.358000 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.359011 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.360653 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:07:43.361160 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:07:43.361311 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:07:43.366267 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:07:43.366280 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:07:43.366299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:07:43.366339 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:07:43.369985 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.370782 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.370789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.370791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.370960 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:07:43.371897 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:07:43.371903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:07:43.371906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:07:43.372353 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:07:43.372689 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0621 07:07:43.373045 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 07:07:43.373312 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:07:43.374572 17767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:43.514359 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17644
I20260621 07:07:43.536573 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17283
I20260621 07:07:43.538112 17345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:07:43.673013 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17283
I20260621 07:08:03.698812 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--webserver_interface=127.1.188.254
--webserver_port=33827
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:04.085685 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:04.086040 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:04.086236 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:04.094172 17825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:08:04.094296 17825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:08:04.094383 17825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:08:04.094468 17825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:04.094591 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:04.094720 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:04.094810 17825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:04.094899 17825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:04.105849 17825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=33827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:04.107547 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:04.109148 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:04.136014 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:04.136157 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:04.136085 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:04.137630 17825 server_base.cc:1034] running on GCE node
I20260621 07:08:04.138695 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:04.140599 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:04.141903 17825 hybrid_clock.cc:648] HybridClock initialized: now 1782025684141820 us; error 68 us; skew 500 ppm
I20260621 07:08:04.142351 17825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:04 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:04.150164 17825 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:08:04.152235 17825 webserver.cc:460] Webserver started at http://127.1.188.254:33827/ using document root <none> and password file <none>
I20260621 07:08:04.153177 17825 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:04.153324 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:04.160430 17825 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:08:04.164013 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:04.165717 17825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260621 07:08:04.165933 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "d1389a0f7a7449a681f56fc7ff5137cc"
format_stamp: "Formatted at 2026-06-21 07:07:40 on dist-test-slave-hlq1"
server_key: "aee35f6ba1fb8550ebb982f3b94d31cd"
server_key_iv: "e16af1d833147d1b43b70a7e3534d124"
server_key_version: "encryptionkey@0"
I20260621 07:08:04.167146 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:04.179939 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:04.184823 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:04.185173 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:04.208107 17825 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42059
I20260621 07:08:04.208166 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42059 every 8 connection(s)
I20260621 07:08:04.209991 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:04.217900 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17825
I20260621 07:08:04.218843 17893 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:08:04.218989 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:33235
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=36363
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:08:04.224424 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap starting.
I20260621 07:08:04.247025 17893 log.cc:824] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:04.261113 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:08:04.261865 17893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap complete.
I20260621 07:08:04.271941 17893 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:04.272940 17893 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Initialized, Role: FOLLOWER
I20260621 07:08:04.273756 17893 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:04.274013 17893 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:04.274176 17893 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:04.274382 17893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 1 FOLLOWER]: Advancing to term 2
I20260621 07:08:04.278719 17893 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:04.279385 17893 leader_election.cc:302] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1389a0f7a7449a681f56fc7ff5137cc; no voters:
I20260621 07:08:04.280143 17893 leader_election.cc:288] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 07:08:04.280277 17897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Leader election won for term 2
I20260621 07:08:04.282092 17897 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 LEADER]: Becoming Leader. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Running, Role: LEADER
I20260621 07:08:04.282727 17897 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:04.282982 17893 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:04.285939 17899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1389a0f7a7449a681f56fc7ff5137cc. Latest consensus state: current_term: 2 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:08:04.286038 17898 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:08:04.286543 17899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:04.286582 17898 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:04.291296 17902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:04.302209 17902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:04.304188 17902 catalog_manager.cc:1259] Loaded cluster ID: 6a7ffb42d24d488487c4d4e2099c6b33
I20260621 07:08:04.304381 17902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:04.310110 17902 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:04.312484 17902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Loaded TSK: 0
I20260621 07:08:04.342181 17914 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:08:04.349813 17902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Generated new TSK 1
I20260621 07:08:04.350209 17902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:08:04.625931 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:04.626274 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:04.626438 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:04.633823 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:04.633970 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:04.634138 17895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:04.644269 17895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:33235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=36363
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:04.645700 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:04.647147 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:04.661216 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:04.661260 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:04.661594 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:04.662236 17895 server_base.cc:1034] running on GCE node
I20260621 07:08:04.662864 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:04.664011 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:04.665436 17895 hybrid_clock.cc:648] HybridClock initialized: now 1782025684665359 us; error 61 us; skew 500 ppm
I20260621 07:08:04.665848 17895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:04 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:04.669466 17895 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:08:04.671070 17895 webserver.cc:460] Webserver started at http://127.1.188.193:36363/ using document root <none> and password file <none>
I20260621 07:08:04.671904 17895 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:04.672057 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:04.677772 17895 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:08:04.680878 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:04.682286 17895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:08:04.682482 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "35bf04323879438dbb46f95969b638dd"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "73a8265f20dfdaf6ea999b8647cba5fc"
server_key_iv: "bf628e73d3a546a56276d623a23d0a18"
server_key_version: "encryptionkey@0"
I20260621 07:08:04.683531 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:04.698585 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:04.702457 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:04.702736 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:04.703716 17895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:04.705029 17895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:04.705156 17895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:04.705282 17895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:04.705379 17895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:04.740213 17895 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33235
I20260621 07:08:04.740326 18042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33235 every 8 connection(s)
I20260621 07:08:04.741722 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:04.750187 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 17895
I20260621 07:08:04.751040 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:44025
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=40485
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:08:04 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025684, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:04.766803 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:04.744881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:36193 (local address 127.1.188.254:42059)
0621 07:08:04.745472 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:04.745484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:04.746271 (+ 787us) server_negotiation.cc:407] Connection header received
0621 07:08:04.748448 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:04.748462 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:04.748793 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:04.749083 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:04.750549 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:04.752705 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:04.754279 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:04.755369 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:04.755598 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:04.759068 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:04.759076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:04.759091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:04.759128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:04.762217 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:04.763211 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:04.763218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:04.763223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:04.763379 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:04.763929 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:04.763932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:04.763934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:04.764434 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:04.764584 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:04.765493 (+ 909us) server_negotiation.cc:299] Negotiation successful
0621 07:08:04.765817 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260621 07:08:04.767203 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:04.745157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:36193)
0621 07:08:04.746054 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:04.746106 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:04.747928 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:04.749215 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:04.749222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:04.749748 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:04.750313 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:04.750340 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:04.752867 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:04.752871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:04.754132 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:04.754144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:04.755611 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:04.755616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:04.755801 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:04.756520 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:04.756541 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:04.758791 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:04.762387 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:04.762398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:04.762415 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:04.763040 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:04.763564 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:04.763571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:04.763578 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:04.763809 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:04.764593 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:04.764603 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:04.765014 (+ 411us) client_negotiation.cc:769] Sending connection context
0621 07:08:04.765314 (+ 300us) client_negotiation.cc:240] Negotiation successful
0621 07:08:04.765908 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260621 07:08:04.770181 18043 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:08:04.770511 18043 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:04.771301 18043 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
I20260621 07:08:04.773571 17858 ts_manager.cc:194] Registered new tserver with Master: 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235)
I20260621 07:08:04.775014 17858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:36193
W20260621 07:08:05.085757 18047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:05.086020 18047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:05.086184 18047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:05.093428 18047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:05.093532 18047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:05.093703 18047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:05.103339 18047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:44025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=40485
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:05.104609 18047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:05.105876 18047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:05.118718 18052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:05.118854 18055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:05.119468 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:05.119553 18047 server_base.cc:1034] running on GCE node
I20260621 07:08:05.120173 18047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:05.121176 18047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:05.122419 18047 hybrid_clock.cc:648] HybridClock initialized: now 1782025685122370 us; error 63 us; skew 500 ppm
I20260621 07:08:05.122756 18047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:05 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025685, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:05.125999 18047 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:08:05.127617 18047 webserver.cc:460] Webserver started at http://127.1.188.194:40485/ using document root <none> and password file <none>
I20260621 07:08:05.128320 18047 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:05.128500 18047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:05.134163 18047 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260621 07:08:05.137120 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:05.138460 18047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:08:05.138646 18047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "c1e30f69ea5f485ba878b393fe1792c0"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "d2ef70134e22ac4627f3d007c3131d98"
server_key_iv: "9d603e5e657157bcb77cb4fe7ba608c0"
server_key_version: "encryptionkey@0"
I20260621 07:08:05.139772 18047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:05.155941 18047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:05.160032 18047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:05.160336 18047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:05.161353 18047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:05.162731 18047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:05.162875 18047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:05.163007 18047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:05.163107 18047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:05.200129 18047 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:44025
I20260621 07:08:05.200179 18174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:44025 every 8 connection(s)
I20260621 07:08:05.201679 18047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:05.208571 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18047
I20260621 07:08:05.209262 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:36037
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=39093
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 07:08:05 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025685, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:05.226989 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.204345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:39091 (local address 127.1.188.254:42059)
0621 07:08:05.204547 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.204552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.205680 (+ 1128us) server_negotiation.cc:407] Connection header received
0621 07:08:05.207751 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.207756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.207827 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.207912 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:05.209886 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.211158 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.213502 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.214311 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.214468 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.219098 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:05.219108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:05.219116 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:05.219167 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:05.223141 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.224129 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.224136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.224139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.224289 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.224819 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.224826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.224832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.225280 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:05.225453 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.226230 (+ 777us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.226527 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:08:05.227681 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.204819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:39091)
0621 07:08:05.205515 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.205557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.207389 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.208179 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.208190 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.208915 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:05.209697 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.209720 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.211339 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.211345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.213326 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.213341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.214548 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.214558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.214840 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.215691 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:05.215719 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:05.218748 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:05.223322 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:05.223332 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:05.223349 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:05.223980 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:05.224412 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:05.224417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:05.224421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:05.224673 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:05.225427 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:05.225435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:05.225854 (+ 419us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.226082 (+ 228us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.226496 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260621 07:08:05.229262 18175 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:08:05.229631 18175 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:05.230597 18175 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
I20260621 07:08:05.232353 17858 ts_manager.cc:194] Registered new tserver with Master: c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025)
I20260621 07:08:05.233160 17858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:39091
W20260621 07:08:05.570606 18178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:05.571075 18178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:05.571225 18178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:05.579223 18178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:05.579345 18178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:05.579540 18178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:05.590306 18178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:36037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=39093
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:05.591925 18178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:05.593439 18178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:05.608515 18183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:05.608466 18184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:05.608417 18186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:05.609462 18178 server_base.cc:1034] running on GCE node
I20260621 07:08:05.610132 18178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:05.611270 18178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:05.612532 18178 hybrid_clock.cc:648] HybridClock initialized: now 1782025685612477 us; error 65 us; skew 500 ppm
I20260621 07:08:05.612897 18178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:05 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025685, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:05.616545 18178 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:08:05.618340 18178 webserver.cc:460] Webserver started at http://127.1.188.195:39093/ using document root <none> and password file <none>
I20260621 07:08:05.619182 18178 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:05.619357 18178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:05.628798 18178 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260621 07:08:05.632021 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:05.633499 18178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:08:05.633699 18178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "c64bc44dcf9a49ae9dc280b8ea641197"
format_stamp: "Formatted at 2026-06-21 07:07:42 on dist-test-slave-hlq1"
server_key: "e7beff4c9fafce2a1142a7cbbafc9a7c"
server_key_iv: "253e2223525e215d9757ea7b4dd6ed41"
server_key_version: "encryptionkey@0"
I20260621 07:08:05.634835 18178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:05.648418 18178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:05.652597 18178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:05.652837 18178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:05.653821 18178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:05.655221 18178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:05.655318 18178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:05.655447 18178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:05.655524 18178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 07:08:05.691535 18178 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:36037
I20260621 07:08:05.691632 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:36037 every 8 connection(s)
I20260621 07:08:05.693276 18178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:05.698283 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18178
Jun 21 07:08:05 dist-test-slave-hlq1 krb5kdc[17270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025685, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:05.716233 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.699251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.1.188.254:42059)
0621 07:08:05.699783 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.699789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.700292 (+ 503us) server_negotiation.cc:407] Connection header received
0621 07:08:05.700437 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.700442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.700541 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.700672 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:05.702476 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.703611 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.705198 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.705907 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.706001 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.709266 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:05.709272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:05.709275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:05.709310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:05.712958 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.713929 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.713942 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.713948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.714134 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.714580 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.714584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.714587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.714886 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:05.715105 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.715552 (+ 447us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.715807 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":412,"thread_start_us":420,"threads_started":1}
I20260621 07:08:05.717797 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.696355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:40691 (local address 127.1.188.254:42059)
0621 07:08:05.696593 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.696598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.697912 (+ 1314us) server_negotiation.cc:407] Connection header received
0621 07:08:05.700886 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.700892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.700969 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.701091 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:05.702914 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.703960 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.705824 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.706462 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.706609 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.710816 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:05.710821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:05.710824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:05.710870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:05.713982 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.714929 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.714934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.714939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.715097 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.715650 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.715654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.715657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.715998 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:05.716150 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.717230 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.717448 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":94}
I20260621 07:08:05.719017 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.696866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:40691)
0621 07:08:05.697695 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.697756 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.700573 (+ 2817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.701349 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.701358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.701916 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:05.702732 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.702759 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.704136 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.704141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.705678 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.705694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.706778 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.706795 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.707166 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.707977 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:05.708003 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:05.710592 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:05.714140 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:05.714150 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:05.714171 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:05.714804 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:05.715225 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:05.715229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:05.715231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:05.715524 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:05.716220 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:05.716227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:05.716704 (+ 477us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.717109 (+ 405us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.717609 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":351,"threads_started":1}
I20260621 07:08:05.720496 18306 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42059
I20260621 07:08:05.720955 18306 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:05.721885 18306 heartbeater.cc:511] Master 127.1.188.254:42059 requested a full tablet report, sending...
I20260621 07:08:05.723685 17858 ts_manager.cc:194] Registered new tserver with Master: c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195:36037)
I20260621 07:08:05.724705 17858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:40691
I20260621 07:08:05.731961 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 07:08:05.740489 17776 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.188.254:42059)
I20260621 07:08:05.740926 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.733238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.1.188.254:42059)
0621 07:08:05.733455 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.733463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.733840 (+ 377us) server_negotiation.cc:407] Connection header received
0621 07:08:05.734131 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.734135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.734289 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.734452 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:05.734948 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.735843 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.737354 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.737804 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.737952 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.738553 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:05.738887 (+ 334us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 07:08:05.739258 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47446: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 07:08:05.741324 18045 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47446: token expired
I20260621 07:08:05.756397 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.741089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.1.188.254:42059)
0621 07:08:05.741280 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.741288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.741596 (+ 308us) server_negotiation.cc:407] Connection header received
0621 07:08:05.741650 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.741655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.741737 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.741825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:05.744064 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.744886 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.746966 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.747384 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.747542 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.750573 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:05.750578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:05.750582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:05.750615 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:05.753533 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.754314 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.754322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.754326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.754470 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:05.754928 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:05.754932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:05.754934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:05.755302 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:05.755600 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.755882 (+ 282us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.756072 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:08:05.760644 17858 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:08:05.768998 17858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:08:05.778167 18043 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
I20260621 07:08:05.815740 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.788990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.1.188.193:33235)
0621 07:08:05.803671 (+ 14681us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.803686 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.805273 (+ 1587us) server_negotiation.cc:407] Connection header received
0621 07:08:05.805660 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.805675 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.806091 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.806335 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:05.807743 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.809335 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.812580 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.814299 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.814466 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.814762 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.815136 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.815322 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14450,"thread_start_us":14218,"threads_started":1}
I20260621 07:08:05.815796 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.788713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33235 (local address 127.0.0.1:41182)
0621 07:08:05.805115 (+ 16402us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.805135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.805403 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.806848 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.806862 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.806907 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:05.807467 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.807485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.809546 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.809553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.812329 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.812345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.814592 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.814605 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.814841 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.814910 (+ 69us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.815059 (+ 149us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.815275 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16224,"mutex_wait_us":1,"thread_start_us":15231,"threads_started":1}
I20260621 07:08:05.817775 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.785186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.1.188.195:36037)
0621 07:08:05.809618 (+ 24432us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.809627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.809668 (+ 41us) server_negotiation.cc:407] Connection header received
0621 07:08:05.809785 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.809793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.810029 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.810252 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:05.810983 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.812235 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.814901 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.816546 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.816708 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.816973 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.817125 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.817322 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24225,"thread_start_us":24008,"threads_started":1}
I20260621 07:08:05.818166 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.784449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44025 (local address 127.0.0.1:49714)
0621 07:08:05.801485 (+ 17036us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.801573 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.801874 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.809640 (+ 7766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.809645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.809675 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:05.810151 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.810162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.812304 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.812316 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.815047 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.815065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.817215 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.817222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.817422 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.817472 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.817546 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.817685 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16574,"mutex_wait_us":4,"thread_start_us":17695,"threads_started":1}
I20260621 07:08:05.819190 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.784730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.1.188.194:44025)
0621 07:08:05.808167 (+ 23437us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.808183 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.808229 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:08:05.808404 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.808417 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.808734 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.809655 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:05.810371 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.812123 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.815262 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.816920 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.818434 (+ 1514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.818677 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.818772 (+ 95us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.818918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23191,"thread_start_us":22965,"threads_started":1}
I20260621 07:08:05.819713 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.784898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:36037 (local address 127.0.0.1:46782)
0621 07:08:05.804279 (+ 19381us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.804303 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.804485 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.810240 (+ 5755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.810245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.810272 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:05.810766 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.810776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.812357 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.812362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.814729 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.814747 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.816775 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.816781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.816953 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.816983 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.817074 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.817216 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19202,"mutex_wait_us":18,"thread_start_us":20982,"threads_started":1}
I20260621 07:08:05.821113 17978 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1abb5372d342009daede8025ae10e9 (DEFAULT_TABLE table=test-table [id=d08f0f34143e4d10a4febe3581563da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:05.823033 17978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1abb5372d342009daede8025ae10e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:05.822803 18110 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1abb5372d342009daede8025ae10e9 (DEFAULT_TABLE table=test-table [id=d08f0f34143e4d10a4febe3581563da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:05.823726 18241 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1abb5372d342009daede8025ae10e9 (DEFAULT_TABLE table=test-table [id=d08f0f34143e4d10a4febe3581563da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:05.824599 18110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1abb5372d342009daede8025ae10e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:05.825204 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1abb5372d342009daede8025ae10e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:05.842546 18322 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Bootstrap starting.
I20260621 07:08:05.844586 18324 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Bootstrap starting.
I20260621 07:08:05.847035 18322 tablet_bootstrap.cc:654] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:05.848407 18322 log.cc:824] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:05.849203 18323 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Bootstrap starting.
I20260621 07:08:05.851194 18324 tablet_bootstrap.cc:654] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:05.852160 18322 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: No bootstrap required, opened a new log
I20260621 07:08:05.852555 18322 ts_tablet_manager.cc:1397] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260621 07:08:05.852639 18324 log.cc:824] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:05.855487 18323 tablet_bootstrap.cc:654] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:05.856349 18324 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: No bootstrap required, opened a new log
I20260621 07:08:05.856652 18324 ts_tablet_manager.cc:1397] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260621 07:08:05.856769 18323 log.cc:824] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:05.861961 18323 tablet_bootstrap.cc:492] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: No bootstrap required, opened a new log
I20260621 07:08:05.862283 18323 ts_tablet_manager.cc:1397] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260621 07:08:05.863755 18322 raft_consensus.cc:348] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.864274 18322 raft_consensus.cc:374] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:05.864413 18322 raft_consensus.cc:729] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e30f69ea5f485ba878b393fe1792c0, State: Initialized, Role: FOLLOWER
I20260621 07:08:05.865273 18322 consensus_queue.cc:260] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.867367 18175 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
I20260621 07:08:05.868001 18324 raft_consensus.cc:348] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.868458 18324 raft_consensus.cc:374] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:05.868616 18324 raft_consensus.cc:729] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bf04323879438dbb46f95969b638dd, State: Initialized, Role: FOLLOWER
I20260621 07:08:05.869469 18322 ts_tablet_manager.cc:1428] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260621 07:08:05.869427 18324 consensus_queue.cc:260] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.872350 18324 ts_tablet_manager.cc:1428] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260621 07:08:05.872632 18323 raft_consensus.cc:348] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.873075 18323 raft_consensus.cc:374] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:05.873205 18323 raft_consensus.cc:729] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64bc44dcf9a49ae9dc280b8ea641197, State: Initialized, Role: FOLLOWER
I20260621 07:08:05.875051 18323 consensus_queue.cc:260] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.876421 18306 heartbeater.cc:503] Master 127.1.188.254:42059 was elected leader, sending a full tablet report...
I20260621 07:08:05.877094 18323 ts_tablet_manager.cc:1428] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260621 07:08:05.941663 18330 raft_consensus.cc:482] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:05.942063 18330 raft_consensus.cc:504] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.944242 18330 leader_election.cc:288] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025), 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235)
W20260621 07:08:05.945220 18307 tablet.cc:2369] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:05.951622 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.944526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:33235 (local address 127.1.188.195:42799)
0621 07:08:05.944746 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.944773 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.944924 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.946046 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.946054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.946087 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:05.946387 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.946402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.947829 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.947835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.949432 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.949445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.950716 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.950723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.950928 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.950967 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.951081 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.951251 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260621 07:08:05.951630 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.945324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:42799 (local address 127.1.188.193:33235)
0621 07:08:05.945602 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.945610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.945636 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:08:05.945738 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.945743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.945829 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.945921 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:05.946588 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.947679 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.949614 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.950553 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.950685 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.950747 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.951158 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.951303 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260621 07:08:05.952046 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.944835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:44025 (local address 127.1.188.195:48569)
0621 07:08:05.945787 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:05.945807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:05.945997 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:05.946566 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:05.946570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:05.946596 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:05.946989 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.947004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.948321 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.948324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.949851 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:05.949865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.951232 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.951238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:05.951438 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.951475 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 07:08:05.951570 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:08:05.951711 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":550,"threads_started":1}
I20260621 07:08:05.952060 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:05.944730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:48569 (local address 127.1.188.194:44025)
0621 07:08:05.945002 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:05.945011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:05.945936 (+ 925us) server_negotiation.cc:407] Connection header received
0621 07:08:05.946321 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:05.946328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:05.946425 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:05.946534 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:05.947210 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.948165 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.950021 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:05.951018 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:05.951181 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:05.951252 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:05.951622 (+ 370us) server_negotiation.cc:299] Negotiation successful
0621 07:08:05.951799 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 07:08:05.953209 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1abb5372d342009daede8025ae10e9" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e30f69ea5f485ba878b393fe1792c0" is_pre_election: true
I20260621 07:08:05.953195 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1abb5372d342009daede8025ae10e9" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35bf04323879438dbb46f95969b638dd" is_pre_election: true
I20260621 07:08:05.953804 17998 raft_consensus.cc:2454] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 0.
I20260621 07:08:05.953846 18130 raft_consensus.cc:2454] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 0.
W20260621 07:08:05.954049 18176 tablet.cc:2369] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:05.954628 18195 leader_election.cc:302] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bf04323879438dbb46f95969b638dd, c64bc44dcf9a49ae9dc280b8ea641197; no voters:
I20260621 07:08:05.955111 18330 raft_consensus.cc:2781] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:08:05.955272 18330 raft_consensus.cc:482] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:08:05.955389 18330 raft_consensus.cc:3037] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:05.958114 18330 raft_consensus.cc:504] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.958742 18330 leader_election.cc:288] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 election: Requested vote from peers c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025), 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235)
I20260621 07:08:05.959183 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1abb5372d342009daede8025ae10e9" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e30f69ea5f485ba878b393fe1792c0"
I20260621 07:08:05.959270 17998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1abb5372d342009daede8025ae10e9" candidate_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35bf04323879438dbb46f95969b638dd"
I20260621 07:08:05.959458 18130 raft_consensus.cc:3037] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:05.959566 17998 raft_consensus.cc:3037] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:05.961949 18130 raft_consensus.cc:2454] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 1.
I20260621 07:08:05.961968 17998 raft_consensus.cc:2454] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c64bc44dcf9a49ae9dc280b8ea641197 in term 1.
I20260621 07:08:05.962399 18195 leader_election.cc:302] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bf04323879438dbb46f95969b638dd, c64bc44dcf9a49ae9dc280b8ea641197; no voters:
I20260621 07:08:05.962700 18330 raft_consensus.cc:2781] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:05.963232 18330 raft_consensus.cc:686] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 LEADER]: Becoming Leader. State: Replica: c64bc44dcf9a49ae9dc280b8ea641197, State: Running, Role: LEADER
I20260621 07:08:05.963795 18330 consensus_queue.cc:237] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } }
I20260621 07:08:05.969353 17857 catalog_manager.cc:5495] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 reported cstate change: term changed from 0 to 1, leader changed from <none> to c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c64bc44dcf9a49ae9dc280b8ea641197" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 36037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 } health_report { overall_health: UNKNOWN } } }
W20260621 07:08:05.994366 18044 tablet.cc:2369] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:06.007171 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:06.001727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.1.188.195:36037)
0621 07:08:06.001947 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:06.001952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:06.001973 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:08:06.002081 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:06.002086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:06.002183 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:06.002323 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:06.002720 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.003632 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.005051 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.005450 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.005548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:06.006029 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:06.006675 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:08:06.006789 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:06.006871 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 07:08:06.006941 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:08:06.016633 18130 raft_consensus.cc:1261] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Refusing update from remote peer c64bc44dcf9a49ae9dc280b8ea641197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:06.016633 17998 raft_consensus.cc:1261] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Refusing update from remote peer c64bc44dcf9a49ae9dc280b8ea641197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:06.017769 18332 consensus_queue.cc:1035] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35bf04323879438dbb46f95969b638dd" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 33235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:06.018190 18330 consensus_queue.cc:1035] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e30f69ea5f485ba878b393fe1792c0" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:06.052399 17858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:06.052774 17858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:06.056888 17858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Sending DeleteTablet for 3 replicas of tablet 4d1abb5372d342009daede8025ae10e9
I20260621 07:08:06.058275 18110 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1abb5372d342009daede8025ae10e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:06 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:49714
I20260621 07:08:06.058706 18241 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1abb5372d342009daede8025ae10e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:06 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:46782
I20260621 07:08:06.058707 17978 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1abb5372d342009daede8025ae10e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:06 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:41182
I20260621 07:08:06.059391 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17895
I20260621 07:08:06.071458 18346 tablet_replica.cc:330] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: stopping tablet replica
I20260621 07:08:06.072273 18346 raft_consensus.cc:2229] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:08:06.073436 18346 raft_consensus.cc:2258] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:06.073755 18344 tablet_replica.cc:330] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: stopping tablet replica
I20260621 07:08:06.074481 18344 raft_consensus.cc:2229] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:06.075338 18344 raft_consensus.cc:2258] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:06.078301 18344 ts_tablet_manager.cc:1905] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:06.078840 18345 tablet_replica.cc:330] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: stopping tablet replica
I20260621 07:08:06.079062 18346 ts_tablet_manager.cc:1905] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:06.079365 18345 raft_consensus.cc:2229] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:06.080040 18345 raft_consensus.cc:2258] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:06.085261 18345 ts_tablet_manager.cc:1905] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:06.087132 18344 ts_tablet_manager.cc:1918] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:06.087445 18344 log.cc:1196] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/4d1abb5372d342009daede8025ae10e9
I20260621 07:08:06.088320 18344 ts_tablet_manager.cc:1939] T 4d1abb5372d342009daede8025ae10e9 P c1e30f69ea5f485ba878b393fe1792c0: Deleting consensus metadata
I20260621 07:08:06.088994 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:06.061407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.1.188.193:33235)
0621 07:08:06.061657 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:06.061666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:06.061691 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:08:06.061776 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:06.061785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:06.061896 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:06.062033 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:06.064919 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.066555 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.069533 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.070592 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.070770 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:06.078091 (+ 7321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:06.078115 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:06.078137 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:06.078197 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:06.082968 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.084204 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.084214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.084218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.084514 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.085040 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.085048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.085053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.085636 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:06.086109 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:06.088170 (+ 2061us) server_negotiation.cc:299] Negotiation successful
0621 07:08:06.088465 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 07:08:06.090929 17842 catalog_manager.cc:4841] TS c1e30f69ea5f485ba878b393fe1792c0 (127.1.188.194:44025): tablet 4d1abb5372d342009daede8025ae10e9 (table test-table [id=d08f0f34143e4d10a4febe3581563da7]) successfully deleted
I20260621 07:08:06.092856 18345 ts_tablet_manager.cc:1918] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:06.093261 18345 log.cc:1196] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/4d1abb5372d342009daede8025ae10e9
I20260621 07:08:06.093845 18038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:06.094184 18346 ts_tablet_manager.cc:1918] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:06.094457 18346 log.cc:1196] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/4d1abb5372d342009daede8025ae10e9
I20260621 07:08:06.095168 18346 ts_tablet_manager.cc:1939] T 4d1abb5372d342009daede8025ae10e9 P c64bc44dcf9a49ae9dc280b8ea641197: Deleting consensus metadata
I20260621 07:08:06.098522 17845 catalog_manager.cc:4841] TS c64bc44dcf9a49ae9dc280b8ea641197 (127.1.188.195:36037): tablet 4d1abb5372d342009daede8025ae10e9 (table test-table [id=d08f0f34143e4d10a4febe3581563da7]) successfully deleted
I20260621 07:08:06.220005 18345 ts_tablet_manager.cc:1939] T 4d1abb5372d342009daede8025ae10e9 P 35bf04323879438dbb46f95969b638dd: Deleting consensus metadata
I20260621 07:08:06.222959 17843 catalog_manager.cc:4841] TS 35bf04323879438dbb46f95969b638dd (127.1.188.193:33235): tablet 4d1abb5372d342009daede8025ae10e9 (table test-table [id=d08f0f34143e4d10a4febe3581563da7]) successfully deleted
I20260621 07:08:06.223537 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17895
I20260621 07:08:06.249938 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18047
I20260621 07:08:06.272972 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:06.251064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.1.188.194:44025)
0621 07:08:06.251328 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:06.251336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:06.251365 (+ 29us) server_negotiation.cc:407] Connection header received
0621 07:08:06.251484 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:06.251492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:06.251600 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:06.251738 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:06.253174 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.254506 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.256342 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.257043 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.257183 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:06.264928 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:06.264957 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:06.264989 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:06.265050 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:06.269322 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.270112 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.270125 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.270129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.270312 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.270752 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.270763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.270767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.271491 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:06.271844 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:06.272226 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 07:08:06.272496 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 07:08:06.273963 18170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:06.371459 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18047
I20260621 07:08:06.392397 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18178
I20260621 07:08:06.409803 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:06.393320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.1.188.195:36037)
0621 07:08:06.393596 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:06.393604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:06.393630 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:08:06.393815 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:06.393821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:06.393939 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:06.394092 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:06.395453 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.396688 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.397973 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:06.398679 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:06.398894 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:06.401825 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:06.401844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:06.401867 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:06.401929 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:06.406320 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.407090 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.407102 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.407107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.407332 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:06.408024 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:06.408030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:06.408034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:06.408527 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:06.408853 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:06.409124 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 07:08:06.409398 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260621 07:08:06.410616 18301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:06.538745 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18178
I20260621 07:08:06.560706 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 17825
I20260621 07:08:06.561936 17888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:06.679572 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 17825
I20260621 07:08:26.702972 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--webserver_interface=127.1.188.254
--webserver_port=33827
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:27.065529 18351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:27.065927 18351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:27.066133 18351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:27.073945 18351 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 07:08:27.074055 18351 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 07:08:27.074136 18351 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 07:08:27.074218 18351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:27.074323 18351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:27.074465 18351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:27.074540 18351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:27.074630 18351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:27.085435 18351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=33827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:27.087138 18351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:27.088884 18351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:27.102406 18359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:27.102486 18356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:27.102406 18357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:27.102876 18351 server_base.cc:1034] running on GCE node
I20260621 07:08:27.104094 18351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:27.105713 18351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:27.106986 18351 hybrid_clock.cc:648] HybridClock initialized: now 1782025707106916 us; error 43 us; skew 500 ppm
I20260621 07:08:27.107379 18351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:27 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025707, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:27.117637 18351 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:08:27.119540 18351 webserver.cc:460] Webserver started at http://127.1.188.254:33827/ using document root <none> and password file <none>
I20260621 07:08:27.120385 18351 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:27.120534 18351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:27.126816 18351 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 07:08:27.130092 18366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:27.131552 18351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:08:27.131748 18351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "d1389a0f7a7449a681f56fc7ff5137cc"
format_stamp: "Formatted at 2026-06-21 07:07:40 on dist-test-slave-hlq1"
server_key: "aee35f6ba1fb8550ebb982f3b94d31cd"
server_key_iv: "e16af1d833147d1b43b70a7e3534d124"
server_key_version: "encryptionkey@0"
I20260621 07:08:27.132843 18351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:27.150720 18351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:27.155738 18351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:27.156066 18351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:27.179147 18351 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42059
I20260621 07:08:27.179162 18417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42059 every 8 connection(s)
I20260621 07:08:27.181214 18351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:27.189790 18418 sys_catalog.cc:263] Verifying existing consensus state
I20260621 07:08:27.191565 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18351
I20260621 07:08:27.192745 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:33235
--local_ip_for_outbound_sockets=127.1.188.193
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=36363
--webserver_interface=127.1.188.193
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:08:27.195061 18418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap starting.
I20260621 07:08:27.222139 18418 log.cc:824] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:27.240410 18418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 07:08:27.241130 18418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Bootstrap complete.
I20260621 07:08:27.251943 18418 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:27.252943 18418 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Initialized, Role: FOLLOWER
I20260621 07:08:27.253839 18418 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:27.254151 18418 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:27.254312 18418 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:27.254527 18418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 2 FOLLOWER]: Advancing to term 3
I20260621 07:08:27.259279 18418 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:27.259984 18418 leader_election.cc:302] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1389a0f7a7449a681f56fc7ff5137cc; no voters:
I20260621 07:08:27.260788 18418 leader_election.cc:288] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 07:08:27.260957 18422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 3 FOLLOWER]: Leader election won for term 3
I20260621 07:08:27.262197 18422 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [term 3 LEADER]: Becoming Leader. State: Replica: d1389a0f7a7449a681f56fc7ff5137cc, State: Running, Role: LEADER
I20260621 07:08:27.262907 18422 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } }
I20260621 07:08:27.263352 18418 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:27.266067 18423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:08:27.266523 18423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:27.267127 18424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1389a0f7a7449a681f56fc7ff5137cc. Latest consensus state: current_term: 3 leader_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1389a0f7a7449a681f56fc7ff5137cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42059 } } }
I20260621 07:08:27.267442 18424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:27.269034 18431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:27.293845 18431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:27.296231 18431 catalog_manager.cc:1259] Loaded cluster ID: 6a7ffb42d24d488487c4d4e2099c6b33
I20260621 07:08:27.296437 18431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:27.302312 18431 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:27.304870 18431 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Loaded TSK: 0
I20260621 07:08:27.305202 18431 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1389a0f7a7449a681f56fc7ff5137cc: Loaded TSK: 1
F20260621 07:08:27.306896 18431 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7468b3c9ad google::LogMessage::Fail() at ??:0
@ 0x7f7468b3f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7468b3c4de google::LogMessage::Flush() at ??:0
@ 0x7f7468b3da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f746c8a07d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f746c88f73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260621 07:08:27.335960 18440 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f747e56b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f747e56a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f747e5f62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f747e5f60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f747e6405e9 std::function<>::operator()() at ??:0
@ 0x7f747e56ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f747e56cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f747e5f4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f747e5f4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f746b788345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f746b78b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f746b78b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f746b766eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f746ea406db start_thread at ??:0
@ 0x7f74679d661f clone at ??:0
W20260621 07:08:27.665186 18420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:27.665526 18420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:27.665701 18420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:27.673846 18420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:27.673998 18420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:27.674216 18420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:27.685132 18420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:33235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=36363
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:27.686947 18420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:27.688609 18420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:27.703661 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:27.703661 18468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:27.703707 18466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:27.704981 18420 server_base.cc:1034] running on GCE node
I20260621 07:08:27.705710 18420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:27.706792 18420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:27.708029 18420 hybrid_clock.cc:648] HybridClock initialized: now 1782025707707964 us; error 59 us; skew 500 ppm
I20260621 07:08:27.708401 18420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:27 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025707, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:27.714949 18420 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:08:27.716723 18420 webserver.cc:460] Webserver started at http://127.1.188.193:36363/ using document root <none> and password file <none>
I20260621 07:08:27.717526 18420 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:27.717638 18420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:27.723948 18420 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260621 07:08:27.726948 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:27.728544 18420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:27.728726 18420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "35bf04323879438dbb46f95969b638dd"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "73a8265f20dfdaf6ea999b8647cba5fc"
server_key_iv: "bf628e73d3a546a56276d623a23d0a18"
server_key_version: "encryptionkey@0"
I20260621 07:08:27.729818 18420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:27.768726 18420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:27.774998 18420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:27.775342 18420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:27.776638 18420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:27.778681 18420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:27.778793 18420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:27.778904 18420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:27.778980 18420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:27.824788 18420 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:33235
I20260621 07:08:27.824856 18587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:33235 every 8 connection(s)
I20260621 07:08:27.826673 18420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:27.830044 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18420
I20260621 07:08:27.830777 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:44025
--local_ip_for_outbound_sockets=127.1.188.194
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=40485
--webserver_interface=127.1.188.194
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:08:27.833799 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:27.831293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:55743)
0621 07:08:27.832351 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:27.832726 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":392,"threads_started":1}
W20260621 07:08:27.834720 18588 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:42059 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:42059: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
I20260621 07:08:27.835807 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:27.835444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:51719)
0621 07:08:27.835603 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:27.835690 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260621 07:08:27.836797 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:27.836443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:45781)
0621 07:08:27.836593 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:27.836679 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 07:08:27.837073 18588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 07:08:28.228032 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:28.228467 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:28.228670 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:28.237242 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:28.237421 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:28.237651 18591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:28.250375 18591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:44025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=40485
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:28.252322 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:28.253993 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:28.269560 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:28.269634 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:28.269609 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:28.270649 18591 server_base.cc:1034] running on GCE node
I20260621 07:08:28.271513 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:28.272684 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:28.273948 18591 hybrid_clock.cc:648] HybridClock initialized: now 1782025708273873 us; error 60 us; skew 500 ppm
I20260621 07:08:28.274343 18591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:28 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025708, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:28.278362 18591 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:08:28.280402 18591 webserver.cc:460] Webserver started at http://127.1.188.194:40485/ using document root <none> and password file <none>
I20260621 07:08:28.281216 18591 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:28.281349 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:28.288148 18591 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 07:08:28.291808 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.293591 18591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:08:28.293826 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "c1e30f69ea5f485ba878b393fe1792c0"
format_stamp: "Formatted at 2026-06-21 07:07:41 on dist-test-slave-hlq1"
server_key: "d2ef70134e22ac4627f3d007c3131d98"
server_key_iv: "9d603e5e657157bcb77cb4fe7ba608c0"
server_key_version: "encryptionkey@0"
I20260621 07:08:28.295076 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:28.314673 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:28.321247 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:28.321668 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:28.323035 18591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:28.324801 18591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:28.324908 18591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.325076 18591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:28.325186 18591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.363338 18591 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:44025
I20260621 07:08:28.363453 18718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:44025 every 8 connection(s)
I20260621 07:08:28.365197 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:28.371174 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.368976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:60161)
0621 07:08:28.369846 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.370110 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":335,"threads_started":1}
W20260621 07:08:28.372000 18719 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:42059 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:42059: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
I20260621 07:08:28.373338 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.372851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:52823)
0621 07:08:28.373096 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.373187 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260621 07:08:28.374490 18721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.374036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:52949)
0621 07:08:28.374216 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.374322 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:08:28.374455 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18591
W20260621 07:08:28.374845 18719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:08:28.375342 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:36037
--local_ip_for_outbound_sockets=127.1.188.195
--tserver_master_addrs=127.1.188.254:42059
--webserver_port=39093
--webserver_interface=127.1.188.195
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:28.762725 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:28.763057 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:28.763164 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:28.771742 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:28.771899 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:28.772046 18722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:28.783622 18722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:36037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=39093
--tserver_master_addrs=127.1.188.254:42059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:28.785226 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:28.786816 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:28.802336 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:28.802294 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:28.802902 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:28.802853 18722 server_base.cc:1034] running on GCE node
I20260621 07:08:28.803735 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:28.804803 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:28.806059 18722 hybrid_clock.cc:648] HybridClock initialized: now 1782025708806003 us; error 43 us; skew 500 ppm
I20260621 07:08:28.806489 18722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:28 dist-test-slave-hlq1 krb5kdc[17270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025708, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:28.810289 18722 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:08:28.812153 18722 webserver.cc:460] Webserver started at http://127.1.188.195:39093/ using document root <none> and password file <none>
I20260621 07:08:28.813011 18722 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:28.813179 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:28.819923 18722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 07:08:28.823297 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.825079 18722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:28.825289 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "c64bc44dcf9a49ae9dc280b8ea641197"
format_stamp: "Formatted at 2026-06-21 07:07:42 on dist-test-slave-hlq1"
server_key: "e7beff4c9fafce2a1142a7cbbafc9a7c"
server_key_iv: "253e2223525e215d9757ea7b4dd6ed41"
server_key_version: "encryptionkey@0"
I20260621 07:08:28.826449 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:28.838822 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.837804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:53597)
0621 07:08:28.838427 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.838541 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":325,"threads_started":1}
I20260621 07:08:28.845237 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:28.850210 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:28.850557 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:28.851976 18722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:28.853701 18722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:28.853838 18722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.853989 18722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:28.854096 18722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:28.891978 18722 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:36037
I20260621 07:08:28.892043 18850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:36037 every 8 connection(s)
I20260621 07:08:28.893939 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:28.898900 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18722
I20260621 07:08:28.900520 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.897755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:58245)
0621 07:08:28.898716 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.898996 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":414,"threads_started":1}
W20260621 07:08:28.901541 18851 heartbeater.cc:644] Failed to heartbeat to 127.1.188.254:42059 (0 consecutive failures): Network error: Failed to ping master at 127.1.188.254:42059: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
I20260621 07:08:28.902472 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.902115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:59093)
0621 07:08:28.902271 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.902348 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260621 07:08:28.903337 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:28.903001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:59583)
0621 07:08:28.903143 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:28.903231 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 07:08:28.903592 18851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 07:08:29.376543 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:29.375620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:47725)
0621 07:08:29.376145 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:29.376271 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
I20260621 07:08:29.840871 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:29.839966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:33487)
0621 07:08:29.840533 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:29.840646 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
I20260621 07:08:29.905169 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:29.904252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:40403)
0621 07:08:29.904772 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:29.904894 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260621 07:08:30.378562 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:30.377678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:42399)
0621 07:08:30.378272 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:30.378364 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":302,"threads_started":1}
I20260621 07:08:30.842845 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:30.841980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:50137)
0621 07:08:30.842507 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:30.842619 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260621 07:08:30.907073 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:30.906231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:32981)
0621 07:08:30.906771 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:30.906864 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260621 07:08:31.380563 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:31.379621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:51933)
0621 07:08:31.380209 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:31.380315 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":305,"threads_started":1}
I20260621 07:08:31.844892 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:31.844034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:53689)
0621 07:08:31.844582 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:31.844683 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":275,"threads_started":1}
I20260621 07:08:31.908902 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:31.908122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:60145)
0621 07:08:31.908596 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:31.908687 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260621 07:08:32.382483 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:32.381630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:46853)
0621 07:08:32.382175 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:32.382272 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":272,"threads_started":1}
I20260621 07:08:32.846814 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:32.845952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:41251)
0621 07:08:32.846487 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:32.846594 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260621 07:08:32.910972 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:32.909969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:58225)
0621 07:08:32.910636 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:32.910741 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":364,"threads_started":1}
I20260621 07:08:33.384459 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:33.383620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:48545)
0621 07:08:33.384137 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:33.384233 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":262,"threads_started":1}
I20260621 07:08:33.848928 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:33.847958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:56659)
0621 07:08:33.848592 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:33.848721 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":339,"threads_started":1}
I20260621 07:08:33.913136 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:33.912136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:33977)
0621 07:08:33.912784 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:33.912897 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":322,"threads_started":1}
I20260621 07:08:34.386724 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:34.385737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:54195)
0621 07:08:34.386346 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:34.386496 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":361,"threads_started":1}
I20260621 07:08:34.850986 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:34.850039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:35581)
0621 07:08:34.850632 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:34.850727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":265,"threads_started":1}
I20260621 07:08:34.915042 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:34.914157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:49401)
0621 07:08:34.914718 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:34.914824 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":271,"threads_started":1}
I20260621 07:08:35.388854 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:35.387892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:38259)
0621 07:08:35.388499 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:35.388596 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":262,"threads_started":1}
I20260621 07:08:35.852979 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:35.852162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:47643)
0621 07:08:35.852679 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:35.852774 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":276,"threads_started":1}
I20260621 07:08:35.917209 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:35.916217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:43621)
0621 07:08:35.916838 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:35.916938 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":278,"threads_started":1}
I20260621 07:08:36.390889 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:36.389965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:47873)
0621 07:08:36.390576 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:36.390683 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":350,"threads_started":1}
I20260621 07:08:36.855365 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:36.854202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:43885)
0621 07:08:36.854933 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:36.855079 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":318,"threads_started":1}
I20260621 07:08:36.919345 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:36.918442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:41931)
0621 07:08:36.919019 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:36.919120 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
I20260621 07:08:37.392905 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:37.392008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:33787)
0621 07:08:37.392597 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:37.392691 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":291,"threads_started":1}
I20260621 07:08:37.857795 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:37.856783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:44761)
0621 07:08:37.857477 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:37.857582 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":410,"threads_started":1}
I20260621 07:08:37.921341 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:37.920464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:36345)
0621 07:08:37.921022 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:37.921123 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":278,"threads_started":1}
I20260621 07:08:38.394820 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:38.394017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:52549)
0621 07:08:38.394533 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:38.394624 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
I20260621 07:08:38.859823 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:38.858888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:36581)
0621 07:08:38.859455 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:38.859599 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260621 07:08:38.923601 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:38.922619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:44549)
0621 07:08:38.923199 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:38.923309 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":321,"threads_started":1}
I20260621 07:08:39.397114 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:39.396120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:47163)
0621 07:08:39.396655 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:39.396792 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
I20260621 07:08:39.861996 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:39.861038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:40733)
0621 07:08:39.861671 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:39.861783 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":338,"threads_started":1}
I20260621 07:08:39.925860 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:39.924865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:34255)
0621 07:08:39.925479 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:39.925606 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":296,"threads_started":1}
I20260621 07:08:40.399211 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:40.398342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:52155)
0621 07:08:40.398898 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:40.399008 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260621 07:08:40.864527 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:40.863535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:56817)
0621 07:08:40.864166 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:40.864294 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":314,"threads_started":1}
I20260621 07:08:40.928196 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:40.927179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:43581)
0621 07:08:40.927892 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:40.927996 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
I20260621 07:08:41.401283 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:41.400294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:40813)
0621 07:08:41.400898 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:41.401014 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
I20260621 07:08:41.866571 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:41.865644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:47107)
0621 07:08:41.866256 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:41.866351 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":369,"threads_started":1}
I20260621 07:08:41.930413 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:41.929451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:51201)
0621 07:08:41.930061 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:41.930185 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":298,"threads_started":1}
I20260621 07:08:42.403213 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:42.402304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:34603)
0621 07:08:42.402878 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:42.402970 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":305,"threads_started":1}
I20260621 07:08:42.868813 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:42.867972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:56031)
0621 07:08:42.868473 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:42.868594 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260621 07:08:42.932708 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:42.931668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:57705)
0621 07:08:42.932302 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:42.932417 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":362,"threads_started":1}
I20260621 07:08:43.405611 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:43.404743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.194:47461)
0621 07:08:43.405256 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:43.405363 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
I20260621 07:08:43.870738 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:43.869900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.193:33291)
0621 07:08:43.870444 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:43.870547 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":303,"threads_started":1}
I20260621 07:08:43.907106 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18420
I20260621 07:08:43.932197 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:43.908585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.1.188.193:33235)
0621 07:08:43.909499 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:43.909529 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:43.909607 (+ 78us) server_negotiation.cc:407] Connection header received
0621 07:08:43.911646 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:43.911657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:43.912206 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:43.912628 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:43.914299 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:43.915710 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:43.917063 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:43.917803 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:43.918040 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:43.924742 (+ 6702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:43.924753 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:43.924774 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:43.924828 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:43.928216 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:43.929003 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:43.929023 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:43.929031 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:43.929222 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:43.929631 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:43.929638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:43.929642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:43.930356 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:43.930708 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:43.931299 (+ 591us) server_negotiation.cc:299] Negotiation successful
0621 07:08:43.931827 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":455,"threads_started":1}
I20260621 07:08:43.934206 18583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:43.938651 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:43.934152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42059 (local address 127.1.188.195:55965)
0621 07:08:43.938323 (+ 4171us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:43.938436 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.254:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4011,"thread_start_us":3871,"threads_started":1}
I20260621 07:08:44.028579 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18420
I20260621 07:08:44.048835 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18591
I20260621 07:08:44.067916 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:44.049932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.1.188.194:44025)
0621 07:08:44.050558 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:44.050576 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:44.050619 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:08:44.052519 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:44.052532 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:44.053081 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:44.053425 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:44.054937 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:44.056406 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:44.057689 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:44.058416 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:44.058559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:44.061452 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:44.061460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:44.061477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:44.061514 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:44.064551 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:44.065246 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:44.065253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:44.065259 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:44.065430 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:44.065784 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:44.065787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:44.065789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:44.066356 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:44.066609 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:44.067012 (+ 403us) server_negotiation.cc:299] Negotiation successful
0621 07:08:44.067580 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":311,"threads_started":1}
I20260621 07:08:44.069584 18714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:44.164677 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18591
I20260621 07:08:44.185600 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18722
I20260621 07:08:44.206425 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:44.186847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.1.188.195:36037)
0621 07:08:44.187618 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:44.187651 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:44.187703 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:08:44.189729 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:44.189741 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:44.190286 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:44.190687 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:44.192379 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:44.193777 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:44.195303 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:44.196167 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:44.196316 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:44.199199 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:44.199208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:44.199229 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:44.199279 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:44.202793 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:44.203654 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:44.203662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:44.203670 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:44.203857 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:44.204262 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:44.204265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:44.204267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:44.204906 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:44.205168 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:44.205553 (+ 385us) server_negotiation.cc:299] Negotiation successful
0621 07:08:44.206064 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":329,"threads_started":1}
I20260621 07:08:44.208202 18846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:44.298039 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18722
2026-06-21T07:08:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65404 ms)
[----------] 30 tests from SecurityITest (305535 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:08:44 dist-test-slave-hlq1 krb5kdc[18908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:08:44 dist-test-slave-hlq1 krb5kdc[18908](info): set up 2 sockets
Jun 21 07:08:44 dist-test-slave-hlq1 krb5kdc[18908](info): commencing operation
krb5kdc: starting...
W20260621 07:08:45.407672 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:08:45 dist-test-slave-hlq1 krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:08:45.650904 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:38691
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:38691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:46.000075 18921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:46.000574 18921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:46.000742 18921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:46.009100 18921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:46.009259 18921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:46.009364 18921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:46.009470 18921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:46.009554 18921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:46.019714 18921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:38691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:38691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:46.021049 18921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:46.022372 18921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:46.034802 18927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:46.034839 18926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:46.034904 18929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:46.035224 18921 server_base.cc:1034] running on GCE node
I20260621 07:08:46.036368 18921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:46.037817 18921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:46.039036 18921 hybrid_clock.cc:648] HybridClock initialized: now 1782025726038972 us; error 47 us; skew 500 ppm
I20260621 07:08:46.039490 18921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:46 dist-test-slave-hlq1 krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025726, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:46.042685 18921 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:08:46.044305 18921 webserver.cc:460] Webserver started at http://127.1.188.254:38695/ using document root <none> and password file <none>
I20260621 07:08:46.044986 18921 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:46.045089 18921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:46.045356 18921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:46.048394 18921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "42f3a3f683f3471994c31011bb81e5c7"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "4624c55d13d20fb2635ae76438fd3fb7"
server_key_iv: "97e86dcd1eb9129055082916c69e7cee"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.049129 18921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "42f3a3f683f3471994c31011bb81e5c7"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "4624c55d13d20fb2635ae76438fd3fb7"
server_key_iv: "97e86dcd1eb9129055082916c69e7cee"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.054497 18921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:08:46.058166 18936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:46.059674 18921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:46.059885 18921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "42f3a3f683f3471994c31011bb81e5c7"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "4624c55d13d20fb2635ae76438fd3fb7"
server_key_iv: "97e86dcd1eb9129055082916c69e7cee"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.060117 18921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:46.086887 18921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:46.091601 18921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:46.091910 18921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:46.111332 18921 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:38691
I20260621 07:08:46.111387 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:38691 every 8 connection(s)
I20260621 07:08:46.112736 18921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:46.114632 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 18921
I20260621 07:08:46.114894 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:08:46.115641 1779 external_mini_cluster.cc:1428] Setting key 6c0eef7739f825984970cd4e12d7159d
I20260621 07:08:46.116844 18988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 07:08:46 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:46.132349 18988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: Bootstrap starting.
I20260621 07:08:46.135792 18988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:46.136874 18988 log.cc:824] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:46.140129 18988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: No bootstrap required, opened a new log
I20260621 07:08:46.140413 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:46.117564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.1.188.254:38691)
0621 07:08:46.118349 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:46.118361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:46.118401 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:08:46.120109 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:46.120131 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:46.120534 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:46.120875 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:46.122876 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.124363 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.126277 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.126993 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.127129 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:46.132530 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:46.132542 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:46.132563 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:46.132611 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:46.135842 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:46.136647 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:46.136654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:46.136660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:46.136848 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:46.137305 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:46.137311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:46.137315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:46.137864 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:46.138276 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:46.138730 (+ 454us) server_negotiation.cc:299] Negotiation successful
0621 07:08:46.139208 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":365,"threads_started":1}
I20260621 07:08:46.148866 18988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } }
I20260621 07:08:46.149384 18988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:46.149523 18988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f3a3f683f3471994c31011bb81e5c7, State: Initialized, Role: FOLLOWER
I20260621 07:08:46.150162 18988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } }
I20260621 07:08:46.150457 18988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:46.150621 18988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:46.150847 18988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:46.153394 18988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } }
I20260621 07:08:46.153971 18988 leader_election.cc:302] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f3a3f683f3471994c31011bb81e5c7; no voters:
I20260621 07:08:46.154580 18988 leader_election.cc:288] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:08:46.154718 18993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:46.155630 18993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [term 1 LEADER]: Becoming Leader. State: Replica: 42f3a3f683f3471994c31011bb81e5c7, State: Running, Role: LEADER
I20260621 07:08:46.156196 18993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } }
I20260621 07:08:46.157060 18988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:46.159135 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f3a3f683f3471994c31011bb81e5c7. Latest consensus state: current_term: 1 leader_uuid: "42f3a3f683f3471994c31011bb81e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } } }
I20260621 07:08:46.159680 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:46.160877 18994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f3a3f683f3471994c31011bb81e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f3a3f683f3471994c31011bb81e5c7" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38691 } } }
I20260621 07:08:46.161252 18994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:46.162026 18999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:46.167820 18999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:46.177000 18999 catalog_manager.cc:1347] Generated new cluster ID: 9552dc37b45d4500817a00acd182504d
I20260621 07:08:46.177142 18999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:46.187446 18999 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:08:46.188809 18999 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:46.196044 18999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: Generated new TSK 0
I20260621 07:08:46.196938 18999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:08:46.263942 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:46.606412 19014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:46.606690 19014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:46.606858 19014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:46.614076 19014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:46.614192 19014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:46.614377 19014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:46.623955 19014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:46.625192 19014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:46.626703 19014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:46.640182 19022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:46.640436 19020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:46.640172 19019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:46.644929 19014 server_base.cc:1034] running on GCE node
I20260621 07:08:46.645548 19014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:46.646510 19014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:46.647754 19014 hybrid_clock.cc:648] HybridClock initialized: now 1782025726647690 us; error 74 us; skew 500 ppm
I20260621 07:08:46.648090 19014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:46 dist-test-slave-hlq1 krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025726, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:46.651535 19014 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:08:46.653087 19014 webserver.cc:460] Webserver started at http://127.1.188.193:45397/ using document root <none> and password file <none>
I20260621 07:08:46.653760 19014 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:46.653858 19014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:46.654105 19014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:46.657086 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "b745dd648a4841c293dd6e942e1e645c"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "b6687ee12cd4637784f76abe95907a53"
server_key_iv: "e840a8529bfcafc2233ad331ef5e5e6d"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.657963 19014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "b745dd648a4841c293dd6e942e1e645c"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "b6687ee12cd4637784f76abe95907a53"
server_key_iv: "e840a8529bfcafc2233ad331ef5e5e6d"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.663250 19014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:08:46.667016 19029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:46.668561 19014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:46.668761 19014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "b745dd648a4841c293dd6e942e1e645c"
format_stamp: "Formatted at 2026-06-21 07:08:46 on dist-test-slave-hlq1"
server_key: "b6687ee12cd4637784f76abe95907a53"
server_key_iv: "e840a8529bfcafc2233ad331ef5e5e6d"
server_key_version: "encryptionkey@0"
I20260621 07:08:46.668972 19014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:46.687945 19014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:46.692552 19014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:46.692876 19014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:46.693895 19014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:46.695485 19014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:46.695588 19014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:46.695750 19014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:46.695807 19014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:46.733341 19014 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:37677
I20260621 07:08:46.733389 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:37677 every 8 connection(s)
I20260621 07:08:46.734735 19014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:46.743762 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19014
I20260621 07:08:46.743983 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:08:46.744648 1779 external_mini_cluster.cc:1428] Setting key 9c4254cb06fe495daedd4094bfba5079
Jun 21 07:08:46 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025726, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:46.755681 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:46.737412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:58199 (local address 127.1.188.254:38691)
0621 07:08:46.737901 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:46.737905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:46.738645 (+ 740us) server_negotiation.cc:407] Connection header received
0621 07:08:46.740657 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:46.740663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:46.740782 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:46.740904 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:46.742394 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.743123 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.744934 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.745334 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.745482 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:46.749378 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:46.749383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:46.749386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:46.749419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:46.752277 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:46.753087 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:46.753094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:46.753098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:46.753248 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:46.753753 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:46.753763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:46.753766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:46.754045 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:46.754232 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:46.754973 (+ 741us) server_negotiation.cc:299] Negotiation successful
0621 07:08:46.755240 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":262,"threads_started":1}
I20260621 07:08:46.756705 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:46.737760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38691 (local address 127.1.188.193:58199)
0621 07:08:46.738478 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:46.738520 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:46.740379 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:46.741146 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:46.741153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:46.741674 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:46.742215 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:46.742233 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.743330 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.743335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:46.744786 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:46.744801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:46.745575 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:46.745584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:46.745851 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:46.746608 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:46.746635 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:46.749135 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:46.752441 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:46.752452 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:46.752470 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:46.752939 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:46.753382 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:46.753388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:46.753392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:46.753596 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:46.754189 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:46.754197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:46.754562 (+ 365us) client_negotiation.cc:769] Sending connection context
0621 07:08:46.754856 (+ 294us) client_negotiation.cc:240] Negotiation successful
0621 07:08:46.755417 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":308,"threads_started":1}
I20260621 07:08:46.758333 19142 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38691
I20260621 07:08:46.758688 19142 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:46.759593 19142 heartbeater.cc:511] Master 127.1.188.254:38691 requested a full tablet report, sending...
I20260621 07:08:46.761862 18953 ts_manager.cc:194] Registered new tserver with Master: b745dd648a4841c293dd6e942e1e645c (127.1.188.193:37677)
I20260621 07:08:46.763568 18953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:58199
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:08:46.800895 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:47.138909 19148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:47.139281 19148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:47.139508 19148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:47.147684 19148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:47.147837 19148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:47.148031 19148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:47.158066 19148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:47.159462 19148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:47.160738 19148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:47.173933 19153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:47.173974 19156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:47.174132 19154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:47.175457 19148 server_base.cc:1034] running on GCE node
I20260621 07:08:47.176050 19148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:47.176954 19148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:47.178179 19148 hybrid_clock.cc:648] HybridClock initialized: now 1782025727178122 us; error 51 us; skew 500 ppm
I20260621 07:08:47.178515 19148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:47 dist-test-slave-hlq1 krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025727, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:47.182055 19148 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:08:47.183676 19148 webserver.cc:460] Webserver started at http://127.1.188.194:45085/ using document root <none> and password file <none>
I20260621 07:08:47.184388 19148 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:47.184535 19148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:47.184863 19148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:47.187850 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "30f8911a037a4cf2bf65e76d443eb787"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "6253465b2d5964e219f5ee583c8f0451"
server_key_iv: "157db46ee772818a056d123b851d32f5"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.188687 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "30f8911a037a4cf2bf65e76d443eb787"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "6253465b2d5964e219f5ee583c8f0451"
server_key_iv: "157db46ee772818a056d123b851d32f5"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.194826 19148 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:08:47.198748 19163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.200158 19148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:47.200335 19148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "30f8911a037a4cf2bf65e76d443eb787"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "6253465b2d5964e219f5ee583c8f0451"
server_key_iv: "157db46ee772818a056d123b851d32f5"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.200557 19148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:47.213994 19148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:47.218430 19148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:47.218740 19148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:47.219763 19148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:47.221099 19148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:47.221212 19148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.221329 19148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:47.221417 19148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.256146 19148 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:36233
I20260621 07:08:47.256245 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:36233 every 8 connection(s)
I20260621 07:08:47.257547 19148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:47.265278 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19148
I20260621 07:08:47.265472 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:08:47.266059 1779 external_mini_cluster.cc:1428] Setting key 48796c7107734ec833dfc47216a52e7b
Jun 21 07:08:47 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025727, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:47.280270 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.260128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:56339 (local address 127.1.188.254:38691)
0621 07:08:47.260555 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.260559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.261689 (+ 1130us) server_negotiation.cc:407] Connection header received
0621 07:08:47.263869 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.263873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.263951 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.264049 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:47.265874 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.266887 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.268743 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.269475 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.269617 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.273076 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:47.273085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:47.273093 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:47.273140 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:47.276942 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.277928 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.277931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.277934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.278056 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.278554 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.278558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.278560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.278845 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:47.278956 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.279839 (+ 883us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.280006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":224,"threads_started":1}
I20260621 07:08:47.281755 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.260704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38691 (local address 127.1.188.194:56339)
0621 07:08:47.261526 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:47.261575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:47.263602 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:47.264513 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:47.264521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:47.265053 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:47.265696 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.265715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.267113 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.267119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.268580 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.268593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.269680 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.269684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.269888 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.270588 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:47.270610 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:47.272848 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:47.277116 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:47.277128 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:47.277149 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:47.277800 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:47.278165 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:47.278169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:47.278174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:47.278426 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:47.278969 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:47.278981 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:47.279439 (+ 458us) client_negotiation.cc:769] Sending connection context
0621 07:08:47.279744 (+ 305us) client_negotiation.cc:240] Negotiation successful
0621 07:08:47.280308 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260621 07:08:47.283246 19276 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38691
I20260621 07:08:47.283624 19276 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:47.284372 19276 heartbeater.cc:511] Master 127.1.188.254:38691 requested a full tablet report, sending...
I20260621 07:08:47.285872 18953 ts_manager.cc:194] Registered new tserver with Master: 30f8911a037a4cf2bf65e76d443eb787 (127.1.188.194:36233)
I20260621 07:08:47.286693 18953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:56339
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:08:47.322419 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:47.703465 19282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:47.703864 19282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:47.704043 19282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:47.712178 19282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:47.712361 19282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:47.712563 19282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:47.724030 19282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:47.725829 19282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:47.727583 19282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:47.742945 19287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:47.742986 19290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:47.743633 19288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:47.744151 19282 server_base.cc:1034] running on GCE node
I20260621 07:08:47.744839 19282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:47.745915 19282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:47.747164 19282 hybrid_clock.cc:648] HybridClock initialized: now 1782025727747104 us; error 74 us; skew 500 ppm
I20260621 07:08:47.747591 19282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:47 dist-test-slave-hlq1 krb5kdc[18908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025727, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:47.751032 19282 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:08:47.752717 19282 webserver.cc:460] Webserver started at http://127.1.188.195:33653/ using document root <none> and password file <none>
I20260621 07:08:47.753454 19282 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:47.753563 19282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:47.753868 19282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:47.757205 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "a1c74828760d44aab3c7d9bd6d6057b4"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "bd5b67ecab47faf71636362433bf8f31"
server_key_iv: "12aacf6808b08528f4b041cf269b5a7e"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.758095 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "a1c74828760d44aab3c7d9bd6d6057b4"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "bd5b67ecab47faf71636362433bf8f31"
server_key_iv: "12aacf6808b08528f4b041cf269b5a7e"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.764173 19282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:08:47.767529 19142 heartbeater.cc:503] Master 127.1.188.254:38691 was elected leader, sending a full tablet report...
I20260621 07:08:47.768208 19297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.770112 19282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:08:47.770257 19282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "a1c74828760d44aab3c7d9bd6d6057b4"
format_stamp: "Formatted at 2026-06-21 07:08:47 on dist-test-slave-hlq1"
server_key: "bd5b67ecab47faf71636362433bf8f31"
server_key_iv: "12aacf6808b08528f4b041cf269b5a7e"
server_key_version: "encryptionkey@0"
I20260621 07:08:47.770450 19282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:47.791005 19282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:47.795768 19282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:47.796056 19282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:47.797168 19282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:47.798702 19282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:47.798779 19282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.798874 19282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:47.798929 19282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:47.836004 19282 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:33475
I20260621 07:08:47.836112 19409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:33475 every 8 connection(s)
I20260621 07:08:47.837553 19282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:47.838160 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19282
I20260621 07:08:47.838379 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:08:47.839141 1779 external_mini_cluster.cc:1428] Setting key 97714dc6816dd0dd3c1c1c0e1995a51b
Jun 21 07:08:47 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025727, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:47.862632 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.841033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:52905 (local address 127.1.188.254:38691)
0621 07:08:47.841580 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.841585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.842699 (+ 1114us) server_negotiation.cc:407] Connection header received
0621 07:08:47.844971 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.844977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.845088 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.845224 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:47.846925 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.848062 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.849741 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.850241 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.850390 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.854794 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:47.854799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:47.854803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:47.854837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:47.858199 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.859033 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.859075 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.859079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.859292 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.859909 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.859915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.859920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.860374 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:47.860538 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.861781 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.862082 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":295,"threads_started":1}
I20260621 07:08:47.864470 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.841633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38691 (local address 127.1.188.195:52905)
0621 07:08:47.842528 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:47.842569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:47.844690 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:47.845493 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:47.845503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:47.846064 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:47.846755 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.846776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.848239 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.848243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.849578 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.849591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.850630 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.850642 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.851051 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.851957 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:47.851988 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:47.854559 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:47.858319 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:47.858342 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:47.858357 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:47.858898 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:47.859440 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:47.859446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:47.859451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:47.859772 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:47.860631 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:47.860640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:47.861223 (+ 583us) client_negotiation.cc:769] Sending connection context
0621 07:08:47.861603 (+ 380us) client_negotiation.cc:240] Negotiation successful
0621 07:08:47.862372 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":347,"threads_started":1}
I20260621 07:08:47.866982 19410 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38691
I20260621 07:08:47.867458 19410 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:47.868773 19410 heartbeater.cc:511] Master 127.1.188.254:38691 requested a full tablet report, sending...
I20260621 07:08:47.870927 18952 ts_manager.cc:194] Registered new tserver with Master: a1c74828760d44aab3c7d9bd6d6057b4 (127.1.188.195:33475)
I20260621 07:08:47.872798 18952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:52905
I20260621 07:08:47.872833 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:08:47.899600 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.880223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.1.188.254:38691)
0621 07:08:47.880507 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.880516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.880876 (+ 360us) server_negotiation.cc:407] Connection header received
0621 07:08:47.881154 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.881160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.881270 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.881367 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:47.883505 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.884783 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.886723 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.887621 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.887763 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.891460 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:47.891468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:47.891473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:47.891519 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:47.895700 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.896808 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.896814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.896820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.896981 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:47.897511 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:47.897518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:47.897522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:47.898138 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:47.898446 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.898859 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.899165 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:08:47.905514 18952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:08:47.913344 18952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:08:47.942682 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.928799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.1.188.195:33475)
0621 07:08:47.931497 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.931505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.931545 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:08:47.931671 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.931680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.932013 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.933155 (+ 1142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:47.933567 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.935575 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.939430 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.941227 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.941358 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.941658 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.942047 (+ 389us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.942274 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2245,"threads_started":1}
I20260621 07:08:47.942709 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.928425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:33475 (local address 127.0.0.1:53056)
0621 07:08:47.929749 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:47.929814 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:47.930054 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:47.932287 (+ 2233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:47.932328 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:47.932367 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:47.932879 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.932898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.935732 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.935738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.939139 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.939161 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.941503 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.941509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.941729 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.941804 (+ 75us) client_negotiation.cc:769] Sending connection context
0621 07:08:47.941976 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 07:08:47.942205 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":108,"thread_start_us":1171,"threads_started":1}
I20260621 07:08:47.942777 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.929320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.1.188.194:36233)
0621 07:08:47.930829 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.930843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.931840 (+ 997us) server_negotiation.cc:407] Connection header received
0621 07:08:47.932176 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.932183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.932420 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.932637 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:47.933905 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.936152 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.939767 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.941441 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.941577 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.941765 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.942203 (+ 438us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.942406 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1108,"threads_started":1}
I20260621 07:08:47.943586 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.928643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:36233 (local address 127.0.0.1:39884)
0621 07:08:47.931712 (+ 3069us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:47.931733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:47.931959 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:47.933009 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:47.933017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:47.933051 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:47.933654 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.933670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.936390 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.936399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.939537 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.939557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.941814 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.941820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.942011 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.942039 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:08:47.942934 (+ 895us) client_negotiation.cc:240] Negotiation successful
0621 07:08:47.943103 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2939,"mutex_wait_us":6,"thread_start_us":2214,"threads_started":1}
I20260621 07:08:47.948051 19211 tablet_service.cc:1467] Processing CreateTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 (DEFAULT_TABLE table=test-table [id=e81962b987184f86aab14ecddbe0f904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:47.948051 19345 tablet_service.cc:1467] Processing CreateTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 (DEFAULT_TABLE table=test-table [id=e81962b987184f86aab14ecddbe0f904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:47.949340 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.931980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:37677 (local address 127.0.0.1:43614)
0621 07:08:47.940321 (+ 8341us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:47.940346 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:47.940474 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:47.942430 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:47.942435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:47.942465 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:47.942918 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.942934 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.945295 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.945300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.947065 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:47.947079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.948535 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.948541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:47.948705 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.948745 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:08:47.948844 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:08:47.948997 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8180,"thread_start_us":8583,"threads_started":1}
I20260621 07:08:47.949633 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:47.932662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.1.188.193:37677)
0621 07:08:47.941334 (+ 8672us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:47.941346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:47.941381 (+ 35us) server_negotiation.cc:407] Connection header received
0621 07:08:47.941505 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:47.941518 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:47.941857 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:47.942115 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:47.943130 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.944886 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.947239 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:47.948305 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:47.948497 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:47.948750 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:47.948965 (+ 215us) server_negotiation.cc:299] Negotiation successful
0621 07:08:47.949161 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8436,"thread_start_us":8446,"threads_started":1}
I20260621 07:08:47.949892 19345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ace305a346d476aa2a32ce74aa5d156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:47.949895 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ace305a346d476aa2a32ce74aa5d156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:47.953541 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 (DEFAULT_TABLE table=test-table [id=e81962b987184f86aab14ecddbe0f904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:47.954938 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ace305a346d476aa2a32ce74aa5d156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:47.966600 19429 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Bootstrap starting.
I20260621 07:08:47.968856 19431 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Bootstrap starting.
I20260621 07:08:47.969419 19430 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Bootstrap starting.
I20260621 07:08:47.972926 19430 tablet_bootstrap.cc:654] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:47.972956 19431 tablet_bootstrap.cc:654] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:47.974445 19430 log.cc:824] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:47.974515 19431 log.cc:824] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:47.979158 19431 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: No bootstrap required, opened a new log
I20260621 07:08:47.979573 19431 ts_tablet_manager.cc:1397] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260621 07:08:47.983281 19430 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: No bootstrap required, opened a new log
I20260621 07:08:47.983598 19430 ts_tablet_manager.cc:1397] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260621 07:08:47.984699 19429 tablet_bootstrap.cc:654] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:47.986070 19429 log.cc:824] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:47.992249 19430 raft_consensus.cc:348] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:47.992748 19430 raft_consensus.cc:374] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:47.992493 19431 raft_consensus.cc:348] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:47.992856 19430 raft_consensus.cc:729] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1c74828760d44aab3c7d9bd6d6057b4, State: Initialized, Role: FOLLOWER
I20260621 07:08:47.992913 19431 raft_consensus.cc:374] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:47.993027 19431 raft_consensus.cc:729] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b745dd648a4841c293dd6e942e1e645c, State: Initialized, Role: FOLLOWER
I20260621 07:08:47.993594 19430 consensus_queue.cc:260] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:47.993736 19431 consensus_queue.cc:260] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:47.994520 19429 tablet_bootstrap.cc:492] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: No bootstrap required, opened a new log
I20260621 07:08:47.994819 19429 ts_tablet_manager.cc:1397] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.007s
I20260621 07:08:47.995779 19431 ts_tablet_manager.cc:1428] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260621 07:08:48.004007 19429 raft_consensus.cc:348] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.004426 19410 heartbeater.cc:503] Master 127.1.188.254:38691 was elected leader, sending a full tablet report...
I20260621 07:08:48.004536 19429 raft_consensus.cc:374] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:48.004659 19429 raft_consensus.cc:729] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f8911a037a4cf2bf65e76d443eb787, State: Initialized, Role: FOLLOWER
I20260621 07:08:48.005115 19430 ts_tablet_manager.cc:1428] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260621 07:08:48.005323 19429 consensus_queue.cc:260] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.006598 19276 heartbeater.cc:503] Master 127.1.188.254:38691 was elected leader, sending a full tablet report...
I20260621 07:08:48.007176 19429 ts_tablet_manager.cc:1428] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260621 07:08:48.009923 19277 tablet.cc:2369] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:08:48.089835 19411 tablet.cc:2369] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:48.216754 19435 raft_consensus.cc:482] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:48.217244 19435 raft_consensus.cc:504] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.218616 19436 raft_consensus.cc:482] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:48.218935 19436 raft_consensus.cc:504] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.219626 19435 leader_election.cc:288] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30f8911a037a4cf2bf65e76d443eb787 (127.1.188.194:36233), b745dd648a4841c293dd6e942e1e645c (127.1.188.193:37677)
I20260621 07:08:48.221161 19436 leader_election.cc:288] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1c74828760d44aab3c7d9bd6d6057b4 (127.1.188.195:33475), 30f8911a037a4cf2bf65e76d443eb787 (127.1.188.194:36233)
I20260621 07:08:48.229862 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.220117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:36233 (local address 127.1.188.195:45929)
0621 07:08:48.220379 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:48.220404 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:48.220578 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:48.221152 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:48.221160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:48.221192 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:48.221427 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.221441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.223226 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.223235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.226103 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.226124 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.228689 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.228699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.228995 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.229053 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 07:08:48.229168 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:08:48.229388 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:08:48.230854 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.220313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:45929 (local address 127.1.188.194:36233)
0621 07:08:48.220578 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.220585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.220611 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:08:48.220718 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.220723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.220815 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.220927 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:48.221658 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.222980 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.226366 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.228220 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.228428 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.228570 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.230238 (+ 1668us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.230481 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260621 07:08:48.231366 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.221179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:45241 (local address 127.1.188.193:37677)
0621 07:08:48.222048 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.222056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.222076 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:08:48.222291 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.222296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.222392 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.222465 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:48.223451 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.224861 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.228003 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.229764 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.229935 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.230049 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.230752 (+ 703us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.230964 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707}
I20260621 07:08:48.232369 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f8911a037a4cf2bf65e76d443eb787" is_pre_election: true
I20260621 07:08:48.233300 19231 raft_consensus.cc:2454] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c74828760d44aab3c7d9bd6d6057b4 in term 0.
I20260621 07:08:48.234438 19301 leader_election.cc:302] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30f8911a037a4cf2bf65e76d443eb787, a1c74828760d44aab3c7d9bd6d6057b4; no voters:
I20260621 07:08:48.235180 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.220541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:37677 (local address 127.1.188.195:45241)
0621 07:08:48.221936 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:48.221956 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:48.222083 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:48.222698 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:48.222702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:48.222728 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:48.223209 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.223225 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.225100 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.225108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.227788 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.227808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.230256 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.230266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.230514 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.230577 (+ 63us) client_negotiation.cc:769] Sending connection context
0621 07:08:48.234308 (+ 3731us) client_negotiation.cc:240] Negotiation successful
0621 07:08:48.234567 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":1,"thread_start_us":1338,"threads_started":1}
I20260621 07:08:48.235988 19435 raft_consensus.cc:2781] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:08:48.236194 19435 raft_consensus.cc:482] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:08:48.236333 19435 raft_consensus.cc:3037] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:48.236886 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b745dd648a4841c293dd6e942e1e645c" is_pre_election: true
I20260621 07:08:48.237519 19097 raft_consensus.cc:2454] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1c74828760d44aab3c7d9bd6d6057b4 in term 0.
W20260621 07:08:48.238399 19143 tablet.cc:2369] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:48.240185 19435 raft_consensus.cc:504] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.241155 19435 leader_election.cc:288] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [CANDIDATE]: Term 1 election: Requested vote from peers 30f8911a037a4cf2bf65e76d443eb787 (127.1.188.194:36233), b745dd648a4841c293dd6e942e1e645c (127.1.188.193:37677)
I20260621 07:08:48.241642 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.222490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:41827 (local address 127.1.188.195:33475)
0621 07:08:48.223059 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.223065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.232152 (+ 9087us) server_negotiation.cc:407] Connection header received
0621 07:08:48.232442 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.232450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.232542 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.232601 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:48.233747 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.234888 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.239148 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.240549 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.240716 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.240806 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.241207 (+ 401us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.241378 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396}
I20260621 07:08:48.241775 19231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f8911a037a4cf2bf65e76d443eb787"
I20260621 07:08:48.241941 19097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b745dd648a4841c293dd6e942e1e645c"
I20260621 07:08:48.242077 19231 raft_consensus.cc:3037] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:48.242205 19097 raft_consensus.cc:3037] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:48.242657 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.221580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:33475 (local address 127.1.188.193:41827)
0621 07:08:48.232030 (+ 10450us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:48.232055 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:48.232280 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:48.232889 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:48.232895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:48.232925 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:48.233469 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.233483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.235175 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.235181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.238937 (+ 3756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.238952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.240790 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.240797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.240973 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.241015 (+ 42us) client_negotiation.cc:769] Sending connection context
0621 07:08:48.241845 (+ 830us) client_negotiation.cc:240] Negotiation successful
0621 07:08:48.242029 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10253,"mutex_wait_us":1,"thread_start_us":9564,"threads_started":1}
I20260621 07:08:48.243085 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.222556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:40367 (local address 127.1.188.194:36233)
0621 07:08:48.231653 (+ 9097us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.231659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.232320 (+ 661us) server_negotiation.cc:407] Connection header received
0621 07:08:48.232559 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.232566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.232647 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.232711 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:48.236212 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.237334 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.240147 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.241450 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.241629 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.241696 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.242589 (+ 893us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.242776 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8981,"thread_start_us":13002,"threads_started":1}
I20260621 07:08:48.243944 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.221629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:36233 (local address 127.1.188.193:40367)
0621 07:08:48.232230 (+ 10601us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:48.232247 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:48.232410 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:48.232889 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:48.232894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:48.233012 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:48.235959 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.235979 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.237533 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.237537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.239917 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:48.239934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.242082 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.242088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:48.242358 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.242396 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 07:08:48.243282 (+ 886us) client_negotiation.cc:240] Negotiation successful
0621 07:08:48.243493 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10485,"mutex_wait_us":2434,"spinlock_wait_cycles":33536,"thread_start_us":10280,"threads_started":1}
I20260621 07:08:48.244176 19365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "b745dd648a4841c293dd6e942e1e645c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" is_pre_election: true
I20260621 07:08:48.244813 19365 raft_consensus.cc:2379] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b745dd648a4841c293dd6e942e1e645c in current term 1: Already voted for candidate a1c74828760d44aab3c7d9bd6d6057b4 in this term.
I20260621 07:08:48.245093 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ace305a346d476aa2a32ce74aa5d156" candidate_uuid: "b745dd648a4841c293dd6e942e1e645c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f8911a037a4cf2bf65e76d443eb787" is_pre_election: true
I20260621 07:08:48.246037 19097 raft_consensus.cc:2454] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c74828760d44aab3c7d9bd6d6057b4 in term 1.
I20260621 07:08:48.246245 19033 leader_election.cc:302] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b745dd648a4841c293dd6e942e1e645c; no voters: 30f8911a037a4cf2bf65e76d443eb787, a1c74828760d44aab3c7d9bd6d6057b4
I20260621 07:08:48.246510 19301 leader_election.cc:302] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1c74828760d44aab3c7d9bd6d6057b4, b745dd648a4841c293dd6e942e1e645c; no voters:
I20260621 07:08:48.246774 19231 raft_consensus.cc:2454] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1c74828760d44aab3c7d9bd6d6057b4 in term 1.
I20260621 07:08:48.246820 19435 raft_consensus.cc:2781] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:48.247080 19436 raft_consensus.cc:2726] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 07:08:48.247498 19435 raft_consensus.cc:686] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 LEADER]: Becoming Leader. State: Replica: a1c74828760d44aab3c7d9bd6d6057b4, State: Running, Role: LEADER
I20260621 07:08:48.248054 19435 consensus_queue.cc:237] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } }
I20260621 07:08:48.253921 18953 catalog_manager.cc:5495] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1c74828760d44aab3c7d9bd6d6057b4 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1c74828760d44aab3c7d9bd6d6057b4" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 33475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 } health_report { overall_health: UNKNOWN } } }
I20260621 07:08:48.323529 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.317611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.1.188.195:33475)
0621 07:08:48.317814 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.317819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.317838 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:08:48.318110 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.318115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.318242 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.318390 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:48.318795 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.319674 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.321065 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.321517 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.321621 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.322186 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:48.322841 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:08:48.322977 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.323070 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.323148 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:08:48.333508 19231 raft_consensus.cc:1261] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 1 FOLLOWER]: Refusing update from remote peer a1c74828760d44aab3c7d9bd6d6057b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:48.333583 19097 raft_consensus.cc:1261] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 1 FOLLOWER]: Refusing update from remote peer a1c74828760d44aab3c7d9bd6d6057b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:48.334636 19442 consensus_queue.cc:1035] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b745dd648a4841c293dd6e942e1e645c" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 37677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:48.335067 19435 consensus_queue.cc:1035] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f8911a037a4cf2bf65e76d443eb787" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:48.348435 19445 mvcc.cc:205] Tried to move back new op lower bound from 7299177383241138176 to 7299177382908297216. Current Snapshot: MvccSnapshot[applied={T|T < 7299177383241138176}]
I20260621 07:08:48.363284 18952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:48.363692 18952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:48.368140 18952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42f3a3f683f3471994c31011bb81e5c7: Sending DeleteTablet for 3 replicas of tablet 8ace305a346d476aa2a32ce74aa5d156
I20260621 07:08:48.369444 19345 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:48 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:53056
I20260621 07:08:48.369716 19211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:48 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:39884
I20260621 07:08:48.370213 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ace305a346d476aa2a32ce74aa5d156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:48 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:43614
I20260621 07:08:48.371218 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19014
I20260621 07:08:48.388074 19456 tablet_replica.cc:330] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: stopping tablet replica
I20260621 07:08:48.388765 19456 raft_consensus.cc:2229] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:48.389493 19456 raft_consensus.cc:2258] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:48.390425 19457 tablet_replica.cc:330] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: stopping tablet replica
I20260621 07:08:48.390790 19455 tablet_replica.cc:330] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: stopping tablet replica
I20260621 07:08:48.391332 19455 raft_consensus.cc:2229] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:48.391924 19455 raft_consensus.cc:2258] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:48.392993 19456 ts_tablet_manager.cc:1905] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:48.393999 19455 ts_tablet_manager.cc:1905] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:48.390897 19457 raft_consensus.cc:2229] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:08:48.394991 19457 raft_consensus.cc:2258] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:48.396996 19457 ts_tablet_manager.cc:1905] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:48.400497 19456 ts_tablet_manager.cc:1918] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:48.400694 19456 log.cc:1196] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/8ace305a346d476aa2a32ce74aa5d156
Jun 21 07:08:48 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:08:48.401288 19456 ts_tablet_manager.cc:1939] T 8ace305a346d476aa2a32ce74aa5d156 P b745dd648a4841c293dd6e942e1e645c: Deleting consensus metadata
I20260621 07:08:48.401794 19455 ts_tablet_manager.cc:1918] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:48.402015 19455 log.cc:1196] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/8ace305a346d476aa2a32ce74aa5d156
I20260621 07:08:48.402635 19455 ts_tablet_manager.cc:1939] T 8ace305a346d476aa2a32ce74aa5d156 P 30f8911a037a4cf2bf65e76d443eb787: Deleting consensus metadata
I20260621 07:08:48.403317 18940 catalog_manager.cc:4841] TS b745dd648a4841c293dd6e942e1e645c (127.1.188.193:37677): tablet 8ace305a346d476aa2a32ce74aa5d156 (table test-table [id=e81962b987184f86aab14ecddbe0f904]) successfully deleted
I20260621 07:08:48.404156 19457 ts_tablet_manager.cc:1918] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:48.404369 19457 log.cc:1196] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/8ace305a346d476aa2a32ce74aa5d156
I20260621 07:08:48.404688 18940 catalog_manager.cc:4841] TS 30f8911a037a4cf2bf65e76d443eb787 (127.1.188.194:36233): tablet 8ace305a346d476aa2a32ce74aa5d156 (table test-table [id=e81962b987184f86aab14ecddbe0f904]) successfully deleted
I20260621 07:08:48.404979 19457 ts_tablet_manager.cc:1939] T 8ace305a346d476aa2a32ce74aa5d156 P a1c74828760d44aab3c7d9bd6d6057b4: Deleting consensus metadata
I20260621 07:08:48.406641 18938 catalog_manager.cc:4841] TS a1c74828760d44aab3c7d9bd6d6057b4 (127.1.188.195:33475): tablet 8ace305a346d476aa2a32ce74aa5d156 (table test-table [id=e81962b987184f86aab14ecddbe0f904]) successfully deleted
I20260621 07:08:48.409430 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.375692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.1.188.193:37677)
0621 07:08:48.375970 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.375979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.384780 (+ 8801us) server_negotiation.cc:407] Connection header received
0621 07:08:48.385323 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.385333 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.385465 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.385559 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:48.391033 (+ 5474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.392305 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.395592 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.396242 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.396341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.402645 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:48.402663 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:48.402683 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:48.402739 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:48.406426 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.407332 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.407340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.407343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.407527 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.407962 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.407965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.407967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.408337 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:48.408547 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.408934 (+ 387us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.409101 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:08:48.410145 19137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:48.506739 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19014
I20260621 07:08:48.526091 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19148
Jun 21 07:08:48 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:08:48.544916 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.527117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.1.188.194:36233)
0621 07:08:48.527316 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.527321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.527339 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:08:48.527459 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.527472 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.527567 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.527667 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:48.529298 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.530023 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.532031 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.532488 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.532597 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.538076 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:48.538098 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:48.538120 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:48.538226 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:48.541506 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.542485 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.542493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.542495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.542632 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.543088 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.543092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.543094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.543802 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:48.544052 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.544409 (+ 357us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.544589 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:08:48.545753 19271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:48.638211 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19148
I20260621 07:08:48.661733 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19282
Jun 21 07:08:48 dist-test-slave-hlq1 krb5kdc[18908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:08:48.680462 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:48.662683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.1.188.195:33475)
0621 07:08:48.662990 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:48.663001 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:48.663024 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:08:48.663192 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:48.663199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:48.663293 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:48.663450 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:48.665164 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.666035 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.667712 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:48.668248 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:48.668445 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:48.673750 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:48.673762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:48.673778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:48.673828 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:48.677486 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.678248 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.678256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.678258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.678378 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:48.678790 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:48.678805 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:48.678807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:48.679191 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:48.679544 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:48.679860 (+ 316us) server_negotiation.cc:299] Negotiation successful
0621 07:08:48.680104 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260621 07:08:48.681303 19405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:48.808481 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19282
I20260621 07:08:48.828161 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 18921
I20260621 07:08:48.829499 18983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:48.941426 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 18921
2026-06-21T07:08:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4649 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T07:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:08:49Z Disabled control of system clock
I20260621 07:08:49.023593 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42479
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:08:49.379195 19466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:49.379523 19466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:49.379606 19466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:49.386770 19466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:49.386875 19466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:49.386914 19466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:49.386963 19466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:49.387007 19466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:49.396068 19466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:42479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:49.397874 19466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:49.400187 19466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:49.413686 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:49.413769 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:49.413723 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:49.414669 19466 server_base.cc:1034] running on GCE node
I20260621 07:08:49.415616 19466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:49.417299 19466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:49.418507 19466 hybrid_clock.cc:648] HybridClock initialized: now 1782025729418449 us; error 49 us; skew 500 ppm
I20260621 07:08:49.418931 19466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:49.425472 19466 webserver.cc:460] Webserver started at http://127.1.188.254:37665/ using document root <none> and password file <none>
I20260621 07:08:49.426388 19466 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:49.426517 19466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:49.426904 19466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:49.430325 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "618ed8cdeadb4c5891cc32d741b99806"
format_stamp: "Formatted at 2026-06-21 07:08:49 on dist-test-slave-hlq1"
server_key: "b95b4c3f83907d959dd40fccaa968474"
server_key_iv: "7406bd3716c43fa86402f34960d1228d"
server_key_version: "encryptionkey@0"
I20260621 07:08:49.431252 19466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "618ed8cdeadb4c5891cc32d741b99806"
format_stamp: "Formatted at 2026-06-21 07:08:49 on dist-test-slave-hlq1"
server_key: "b95b4c3f83907d959dd40fccaa968474"
server_key_iv: "7406bd3716c43fa86402f34960d1228d"
server_key_version: "encryptionkey@0"
I20260621 07:08:49.437079 19466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:08:49.440883 19480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:49.442420 19466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:08:49.442658 19466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "618ed8cdeadb4c5891cc32d741b99806"
format_stamp: "Formatted at 2026-06-21 07:08:49 on dist-test-slave-hlq1"
server_key: "b95b4c3f83907d959dd40fccaa968474"
server_key_iv: "7406bd3716c43fa86402f34960d1228d"
server_key_version: "encryptionkey@0"
I20260621 07:08:49.442939 19466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:49.466638 19466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:49.468003 19466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:49.468389 19466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:49.491104 19466 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:42479
I20260621 07:08:49.491127 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:42479 every 8 connection(s)
I20260621 07:08:49.492851 19466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:49.497454 19532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:49.498067 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19466
I20260621 07:08:49.498276 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:08:49.498878 1779 external_mini_cluster.cc:1428] Setting key 93716615a9ba57bfb7fe25e680bcae5e
I20260621 07:08:49.512804 19532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: Bootstrap starting.
I20260621 07:08:49.514683 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:49.501214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.1.188.254:42479)
0621 07:08:49.502423 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:49.502435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:49.502487 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:08:49.504427 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:49.504451 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:49.504972 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:49.505329 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:49.505986 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:49.507610 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:49.509691 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:49.510435 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:49.510578 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:49.512081 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:49.512090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:49.512106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:49.512728 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:49.512823 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:49.512846 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:49.513022 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:49.513258 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:08:49.513616 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":467,"threads_started":1}
I20260621 07:08:49.516762 19532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:49.518086 19532 log.cc:824] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:49.521406 19532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: No bootstrap required, opened a new log
I20260621 07:08:49.530229 19532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } }
I20260621 07:08:49.530695 19532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:49.530800 19532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618ed8cdeadb4c5891cc32d741b99806, State: Initialized, Role: FOLLOWER
I20260621 07:08:49.531647 19532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } }
I20260621 07:08:49.531920 19532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:49.532090 19532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:49.532274 19532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:49.534930 19532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } }
I20260621 07:08:49.535634 19532 leader_election.cc:302] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 618ed8cdeadb4c5891cc32d741b99806; no voters:
I20260621 07:08:49.536337 19532 leader_election.cc:288] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:08:49.536548 19537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:49.537403 19537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [term 1 LEADER]: Becoming Leader. State: Replica: 618ed8cdeadb4c5891cc32d741b99806, State: Running, Role: LEADER
I20260621 07:08:49.538137 19537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } }
I20260621 07:08:49.538975 19532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:49.541412 19538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "618ed8cdeadb4c5891cc32d741b99806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } } }
I20260621 07:08:49.541929 19538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:49.541848 19539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 618ed8cdeadb4c5891cc32d741b99806. Latest consensus state: current_term: 1 leader_uuid: "618ed8cdeadb4c5891cc32d741b99806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618ed8cdeadb4c5891cc32d741b99806" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 42479 } } }
I20260621 07:08:49.542164 19539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:49.544075 19543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:49.550617 19543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:49.560812 19543 catalog_manager.cc:1347] Generated new cluster ID: 4352cdfb8d0c42cab401d2058a611863
I20260621 07:08:49.560977 19543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:49.579314 19543 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:08:49.581521 19543 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:49.592211 19543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: Generated new TSK 0
I20260621 07:08:49.593566 19543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:08:49.607228 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--builtin_ntp_servers=127.1.188.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:08:50.001652 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:50.002115 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:50.002302 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:50.011080 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:50.011320 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:50.011575 19556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:50.024459 19556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:50.026436 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:50.028875 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:50.046038 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:50.045945 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:50.046240 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:50.048209 19556 server_base.cc:1034] running on GCE node
I20260621 07:08:50.049130 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:50.050503 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:50.051891 19556 hybrid_clock.cc:648] HybridClock initialized: now 1782025730051829 us; error 55 us; skew 500 ppm
I20260621 07:08:50.052291 19556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:50.055378 19556 webserver.cc:460] Webserver started at http://127.1.188.193:45197/ using document root <none> and password file <none>
I20260621 07:08:50.056442 19556 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:50.056565 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:50.056928 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:50.060642 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "d37a9d68b7ed4531870bb5079d60d7c1"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "1b82e7c0b3e42bfc9115469ceb946edb"
server_key_iv: "ff996d9d6309790409fc1a80bb806295"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.061609 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "d37a9d68b7ed4531870bb5079d60d7c1"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "1b82e7c0b3e42bfc9115469ceb946edb"
server_key_iv: "ff996d9d6309790409fc1a80bb806295"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.068427 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260621 07:08:50.073057 19570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.074960 19556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260621 07:08:50.075146 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "d37a9d68b7ed4531870bb5079d60d7c1"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "1b82e7c0b3e42bfc9115469ceb946edb"
server_key_iv: "ff996d9d6309790409fc1a80bb806295"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.075384 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:50.103287 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:50.104453 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:50.104801 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:50.106020 19556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:50.108148 19556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:50.108258 19556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.108356 19556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:50.108431 19556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.148658 19556 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:39063
I20260621 07:08:50.148706 19682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:39063 every 8 connection(s)
I20260621 07:08:50.150354 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:50.158165 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19556
I20260621 07:08:50.158393 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:08:50.158970 1779 external_mini_cluster.cc:1428] Setting key 31a8cdea99ce01d6bb3f6cb6c1be44f1
I20260621 07:08:50.162513 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--builtin_ntp_servers=127.1.188.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:08:50.165936 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:50.153514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:34767 (local address 127.1.188.254:42479)
0621 07:08:50.154067 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:50.154072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:50.155358 (+ 1286us) server_negotiation.cc:407] Connection header received
0621 07:08:50.157454 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:50.157460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:50.157564 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:50.157702 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:50.159064 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.160127 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.162482 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.163023 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.163138 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:50.165036 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:50.165041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:50.165044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:50.165065 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:50.165110 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:50.165132 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:50.165223 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:50.165524 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 07:08:50.165620 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":306,"threads_started":1}
I20260621 07:08:50.167562 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:50.154171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42479 (local address 127.1.188.193:34767)
0621 07:08:50.155174 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:50.155213 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:50.157130 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:50.157978 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:50.157987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:50.158096 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:50.158874 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:50.158894 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.160338 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.160345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:50.162283 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:50.162303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.163288 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.163299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:50.163702 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:50.164633 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:50.164638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:50.164701 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:50.164816 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:50.165257 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:50.165272 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:50.165283 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:08:50.165508 (+ 225us) client_negotiation.cc:240] Negotiation successful
0621 07:08:50.165995 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":310,"threads_started":1}
I20260621 07:08:50.169509 19683 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42479
I20260621 07:08:50.169874 19683 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:50.170774 19683 heartbeater.cc:511] Master 127.1.188.254:42479 requested a full tablet report, sending...
I20260621 07:08:50.173286 19497 ts_manager.cc:194] Registered new tserver with Master: d37a9d68b7ed4531870bb5079d60d7c1 (127.1.188.193:39063)
I20260621 07:08:50.175725 19497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.193:34767
W20260621 07:08:50.551614 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:50.551965 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:50.552068 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:50.560259 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:50.560395 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:50.560528 19687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:50.571074 19687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:50.572821 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:50.574995 19687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:50.590372 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:50.590416 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:50.590351 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:50.591293 19687 server_base.cc:1034] running on GCE node
I20260621 07:08:50.592049 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:50.593271 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:50.594496 19687 hybrid_clock.cc:648] HybridClock initialized: now 1782025730594454 us; error 36 us; skew 500 ppm
I20260621 07:08:50.594822 19687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:50.597514 19687 webserver.cc:460] Webserver started at http://127.1.188.194:40377/ using document root <none> and password file <none>
I20260621 07:08:50.598284 19687 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:50.598376 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:50.598686 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:50.602026 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "5969bcf9b8174d658cc25a4e06019ffa"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "7991cbe96954fd29a6e5f68e814c32b4"
server_key_iv: "e3a2c01c818fa4fc25df24f6d477031e"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.602861 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "5969bcf9b8174d658cc25a4e06019ffa"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "7991cbe96954fd29a6e5f68e814c32b4"
server_key_iv: "e3a2c01c818fa4fc25df24f6d477031e"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.608664 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 07:08:50.612752 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.614264 19687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 07:08:50.614410 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "5969bcf9b8174d658cc25a4e06019ffa"
format_stamp: "Formatted at 2026-06-21 07:08:50 on dist-test-slave-hlq1"
server_key: "7991cbe96954fd29a6e5f68e814c32b4"
server_key_iv: "e3a2c01c818fa4fc25df24f6d477031e"
server_key_version: "encryptionkey@0"
I20260621 07:08:50.614621 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:50.630095 19687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:50.630992 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:50.631295 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:50.632378 19687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:50.633971 19687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:50.634063 19687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.634156 19687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:50.634213 19687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:50.672436 19687 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:34415
I20260621 07:08:50.672552 19813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:34415 every 8 connection(s)
I20260621 07:08:50.674085 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:50.677721 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19687
I20260621 07:08:50.677923 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:08:50.678655 1779 external_mini_cluster.cc:1428] Setting key 53bbe1c3437ed7038ccfdca4ab66189e
I20260621 07:08:50.682510 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--builtin_ntp_servers=127.1.188.212:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:08:50.691689 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:50.677358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:46755 (local address 127.1.188.254:42479)
0621 07:08:50.677954 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:50.677982 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:50.679656 (+ 1674us) server_negotiation.cc:407] Connection header received
0621 07:08:50.682032 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:50.682040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:50.682157 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:50.682301 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:50.683845 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.685169 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.687245 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.688080 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.688272 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:50.690404 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:50.690413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:50.690422 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:50.690458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:50.690526 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:50.690549 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:50.690655 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:50.690955 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 07:08:50.691117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":294,"threads_started":1}
I20260621 07:08:50.693351 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:50.678118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42479 (local address 127.1.188.194:46755)
0621 07:08:50.679376 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:50.679482 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:50.681719 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:50.682641 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:50.682657 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:50.682821 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:50.683634 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:50.683657 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.685358 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.685362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:50.686887 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:50.686903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:50.688433 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:50.688443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:50.688837 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:50.689892 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:50.689898 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:50.689970 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:50.690114 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:50.690671 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:50.690683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:50.690695 (+ 12us) client_negotiation.cc:769] Sending connection context
0621 07:08:50.690911 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 07:08:50.691515 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":414,"threads_started":1}
I20260621 07:08:50.695341 19814 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42479
I20260621 07:08:50.695801 19814 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:50.696789 19814 heartbeater.cc:511] Master 127.1.188.254:42479 requested a full tablet report, sending...
I20260621 07:08:50.698805 19497 ts_manager.cc:194] Registered new tserver with Master: 5969bcf9b8174d658cc25a4e06019ffa (127.1.188.194:34415)
I20260621 07:08:50.699865 19497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.194:46755
W20260621 07:08:51.074334 19818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:51.074667 19818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:51.074860 19818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:51.082978 19818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:51.083097 19818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:51.083279 19818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:51.093585 19818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:42479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:51.095328 19818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:51.097607 19818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:51.112816 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:51.113040 19826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:51.112870 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:51.113821 19818 server_base.cc:1034] running on GCE node
I20260621 07:08:51.114552 19818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:51.115718 19818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:51.116933 19818 hybrid_clock.cc:648] HybridClock initialized: now 1782025731116882 us; error 30 us; skew 500 ppm
I20260621 07:08:51.117303 19818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:51.120404 19818 webserver.cc:460] Webserver started at http://127.1.188.195:33927/ using document root <none> and password file <none>
I20260621 07:08:51.121284 19818 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:51.121381 19818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:51.121935 19818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:51.125563 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "a9cdc9c802ad4011877b39e3dfa1d663"
format_stamp: "Formatted at 2026-06-21 07:08:51 on dist-test-slave-hlq1"
server_key: "fb0b8b98d2d4cb7cdba4141c7e92f389"
server_key_iv: "eb99478c468c3215965403c754e8843b"
server_key_version: "encryptionkey@0"
I20260621 07:08:51.126535 19818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "a9cdc9c802ad4011877b39e3dfa1d663"
format_stamp: "Formatted at 2026-06-21 07:08:51 on dist-test-slave-hlq1"
server_key: "fb0b8b98d2d4cb7cdba4141c7e92f389"
server_key_iv: "eb99478c468c3215965403c754e8843b"
server_key_version: "encryptionkey@0"
I20260621 07:08:51.132841 19818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260621 07:08:51.137097 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:51.138914 19818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260621 07:08:51.139128 19818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "a9cdc9c802ad4011877b39e3dfa1d663"
format_stamp: "Formatted at 2026-06-21 07:08:51 on dist-test-slave-hlq1"
server_key: "fb0b8b98d2d4cb7cdba4141c7e92f389"
server_key_iv: "eb99478c468c3215965403c754e8843b"
server_key_version: "encryptionkey@0"
I20260621 07:08:51.139412 19818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:51.161311 19818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:51.162385 19818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:51.162766 19818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:51.164047 19818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:51.165817 19818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:51.165961 19818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:51.166115 19818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:51.166224 19818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:51.180035 19683 heartbeater.cc:503] Master 127.1.188.254:42479 was elected leader, sending a full tablet report...
I20260621 07:08:51.205341 19818 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:45299
I20260621 07:08:51.205408 19944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:45299 every 8 connection(s)
I20260621 07:08:51.207033 19818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:51.207545 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 19818
I20260621 07:08:51.207823 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:08:51.208555 1779 external_mini_cluster.cc:1428] Setting key d121a1b2f8fee156f18e3e3654b8d9a3
I20260621 07:08:51.223167 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.210757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:43755 (local address 127.1.188.254:42479)
0621 07:08:51.211701 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.211707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.213304 (+ 1597us) server_negotiation.cc:407] Connection header received
0621 07:08:51.215384 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.215390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.215531 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.215669 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:51.216941 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.217851 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.219863 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.220304 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.220462 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.222106 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:51.222112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:51.222116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:51.222141 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:51.222190 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:51.222203 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:51.222308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.222567 (+ 259us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.222698 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":656,"threads_started":1}
I20260621 07:08:51.224617 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.211953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:42479 (local address 127.1.188.195:43755)
0621 07:08:51.213129 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.213181 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.215073 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.215946 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.215956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.216066 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:51.216749 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.216767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.218055 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.218061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.219686 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.219700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.220590 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.220598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.220867 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.221701 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:51.221704 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:51.221754 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:51.221860 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:51.222303 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:51.222316 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:51.222327 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.222557 (+ 230us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.223072 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":412,"threads_started":1}
I20260621 07:08:51.226743 19945 heartbeater.cc:347] Connected to a master server at 127.1.188.254:42479
I20260621 07:08:51.227147 19945 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:51.228152 19945 heartbeater.cc:511] Master 127.1.188.254:42479 requested a full tablet report, sending...
I20260621 07:08:51.230391 19497 ts_manager.cc:194] Registered new tserver with Master: a9cdc9c802ad4011877b39e3dfa1d663 (127.1.188.195:45299)
I20260621 07:08:51.231509 19497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.195:43755
I20260621 07:08:51.238487 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:08:51.250559 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.243291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.1.188.254:42479)
0621 07:08:51.243632 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.243641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.244113 (+ 472us) server_negotiation.cc:407] Connection header received
0621 07:08:51.244360 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.244368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.244450 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.244571 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:51.245355 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.246149 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.247888 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.248380 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.248533 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.249768 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:51.249773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:51.249776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:51.249797 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:51.249834 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:51.249847 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:51.249951 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.250121 (+ 170us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.250234 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260621 07:08:51.255187 19497 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:08:51.263068 19497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:08:51.307662 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.280802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:45299 (local address 127.0.0.1:49268)
0621 07:08:51.297408 (+ 16606us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.297426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.297545 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.298549 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.298553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.298578 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:51.299019 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.299033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.301176 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.301181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.303991 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.304004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.306481 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.306487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.306719 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.306775 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.306933 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.307178 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16459,"mutex_wait_us":160,"thread_start_us":15523,"threads_started":1}
I20260621 07:08:51.307729 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.279157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39063 (local address 127.0.0.1:46492)
0621 07:08:51.294952 (+ 15795us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.295031 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.295433 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.300056 (+ 4623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.300060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.300085 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:51.300461 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.300473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.302518 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.302524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.304826 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.304837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.306868 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.306873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.307088 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.307122 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.307215 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.307347 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15263,"thread_start_us":14938,"threads_started":1}
I20260621 07:08:51.307753 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.281707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.1.188.195:45299)
0621 07:08:51.296116 (+ 14409us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.296132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.297666 (+ 1534us) server_negotiation.cc:407] Connection header received
0621 07:08:51.297785 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.297794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.298025 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.298229 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:51.299224 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.300973 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.304207 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.305794 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.305967 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.306341 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.307012 (+ 671us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.307255 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14137,"thread_start_us":13940,"threads_started":1}
I20260621 07:08:51.308800 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.279677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:34415 (local address 127.0.0.1:41338)
0621 07:08:51.296699 (+ 17022us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.296725 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.296885 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.298321 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.298331 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.298368 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:51.298883 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.298903 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.301135 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.301141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.304329 (+ 3188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.304344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.308225 (+ 3881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.308232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.308385 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.308409 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.308463 (+ 54us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.308563 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16862,"mutex_wait_us":118,"thread_start_us":17459,"threads_started":1}
I20260621 07:08:51.308816 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.279627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.1.188.193:39063)
0621 07:08:51.299522 (+ 19895us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.299529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.299555 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:08:51.299651 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.299657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.299838 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.300061 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:51.300725 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.302320 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.305018 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.306351 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.306504 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.306788 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.308213 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.308427 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19655,"thread_start_us":19500,"threads_started":1}
I20260621 07:08:51.310205 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.280325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.1.188.194:34415)
0621 07:08:51.297127 (+ 16802us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.297140 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.297183 (+ 43us) server_negotiation.cc:407] Connection header received
0621 07:08:51.297330 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.297340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.297730 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.297969 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:51.299148 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.300973 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.304569 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.307738 (+ 3169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.309214 (+ 1476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.309521 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.309622 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.309824 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16593,"thread_start_us":16455,"threads_started":1}
I20260621 07:08:51.313076 19880 tablet_service.cc:1467] Processing CreateTablet for tablet ae0a8b20865648ceaf3588b283bb78bf (DEFAULT_TABLE table=test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:51.313081 19618 tablet_service.cc:1467] Processing CreateTablet for tablet ae0a8b20865648ceaf3588b283bb78bf (DEFAULT_TABLE table=test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:51.314670 19880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0a8b20865648ceaf3588b283bb78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:51.314721 19618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0a8b20865648ceaf3588b283bb78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:51.314204 19749 tablet_service.cc:1467] Processing CreateTablet for tablet ae0a8b20865648ceaf3588b283bb78bf (DEFAULT_TABLE table=test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:51.315717 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0a8b20865648ceaf3588b283bb78bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:51.336481 19965 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Bootstrap starting.
I20260621 07:08:51.342322 19965 tablet_bootstrap.cc:654] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:51.344388 19965 log.cc:824] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:51.348340 19964 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Bootstrap starting.
I20260621 07:08:51.350816 19967 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Bootstrap starting.
I20260621 07:08:51.355253 19964 tablet_bootstrap.cc:654] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:51.356947 19964 log.cc:824] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:51.357100 19965 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: No bootstrap required, opened a new log
I20260621 07:08:51.357103 19967 tablet_bootstrap.cc:654] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:51.357467 19965 ts_tablet_manager.cc:1397] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20260621 07:08:51.358497 19967 log.cc:824] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:51.367255 19964 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: No bootstrap required, opened a new log
I20260621 07:08:51.367699 19964 ts_tablet_manager.cc:1397] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.002s
I20260621 07:08:51.375371 19965 raft_consensus.cc:348] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.376103 19965 raft_consensus.cc:374] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:51.376219 19965 raft_consensus.cc:729] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cdc9c802ad4011877b39e3dfa1d663, State: Initialized, Role: FOLLOWER
I20260621 07:08:51.377086 19965 consensus_queue.cc:260] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.379714 19967 tablet_bootstrap.cc:492] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: No bootstrap required, opened a new log
I20260621 07:08:51.380190 19967 ts_tablet_manager.cc:1397] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260621 07:08:51.384701 19964 raft_consensus.cc:348] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.385458 19964 raft_consensus.cc:374] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:51.385632 19964 raft_consensus.cc:729] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5969bcf9b8174d658cc25a4e06019ffa, State: Initialized, Role: FOLLOWER
I20260621 07:08:51.386667 19964 consensus_queue.cc:260] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.388365 19945 heartbeater.cc:503] Master 127.1.188.254:42479 was elected leader, sending a full tablet report...
I20260621 07:08:51.389355 19965 ts_tablet_manager.cc:1428] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Time spent starting tablet: real 0.032s user 0.018s sys 0.011s
I20260621 07:08:51.393256 19964 ts_tablet_manager.cc:1428] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Time spent starting tablet: real 0.025s user 0.019s sys 0.001s
I20260621 07:08:51.393798 19814 heartbeater.cc:503] Master 127.1.188.254:42479 was elected leader, sending a full tablet report...
I20260621 07:08:51.395828 19967 raft_consensus.cc:348] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.396420 19967 raft_consensus.cc:374] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:51.396545 19967 raft_consensus.cc:729] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37a9d68b7ed4531870bb5079d60d7c1, State: Initialized, Role: FOLLOWER
I20260621 07:08:51.397377 19967 consensus_queue.cc:260] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.399693 19967 ts_tablet_manager.cc:1428] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
W20260621 07:08:51.403357 19684 tablet.cc:2369] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:08:51.426695 19815 tablet.cc:2369] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:08:51.459576 19946 tablet.cc:2369] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:51.555321 19970 raft_consensus.cc:482] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:51.555742 19970 raft_consensus.cc:504] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.558499 19970 leader_election.cc:288] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d37a9d68b7ed4531870bb5079d60d7c1 (127.1.188.193:39063), 5969bcf9b8174d658cc25a4e06019ffa (127.1.188.194:34415)
I20260621 07:08:51.566437 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.559253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:41379 (local address 127.1.188.194:34415)
0621 07:08:51.559563 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.559572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.560387 (+ 815us) server_negotiation.cc:407] Connection header received
0621 07:08:51.560590 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.560594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.560729 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.560821 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:51.561619 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.562552 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.564499 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.565331 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.565485 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.565562 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.565979 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.566119 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 07:08:51.566737 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.558873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:34415 (local address 127.1.188.195:41379)
0621 07:08:51.560273 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.560290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.560434 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.560877 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.560885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.560924 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:51.561409 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.561423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.562704 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.562714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.564304 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.564316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.565513 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.565522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.565746 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.565802 (+ 56us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.565916 (+ 114us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.566092 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":26,"thread_start_us":1087,"threads_started":1}
I20260621 07:08:51.567348 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.558798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39063 (local address 127.1.188.195:39585)
0621 07:08:51.559360 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:51.559387 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:51.559587 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:51.560700 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:51.560706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:51.560737 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:51.560974 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.560990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.562353 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.562362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.565050 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:51.565070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.566634 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.566639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:51.566844 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.566877 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 07:08:51.567012 (+ 135us) client_negotiation.cc:240] Negotiation successful
0621 07:08:51.567165 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":65}
I20260621 07:08:51.567922 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.559981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:39585 (local address 127.1.188.193:39063)
0621 07:08:51.560254 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.560263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.560288 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:08:51.560394 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.560400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.560519 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.560652 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:51.561149 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.562199 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.565239 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.566325 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.566476 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.566552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.567576 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.567723 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 07:08:51.567925 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0a8b20865648ceaf3588b283bb78bf" candidate_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5969bcf9b8174d658cc25a4e06019ffa" is_pre_election: true
I20260621 07:08:51.568591 19769 raft_consensus.cc:2454] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9cdc9c802ad4011877b39e3dfa1d663 in term 0.
I20260621 07:08:51.569013 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0a8b20865648ceaf3588b283bb78bf" candidate_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" is_pre_election: true
I20260621 07:08:51.569387 19833 leader_election.cc:302] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5969bcf9b8174d658cc25a4e06019ffa, a9cdc9c802ad4011877b39e3dfa1d663; no voters:
I20260621 07:08:51.569614 19638 raft_consensus.cc:2454] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9cdc9c802ad4011877b39e3dfa1d663 in term 0.
I20260621 07:08:51.569841 19970 raft_consensus.cc:2781] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:08:51.570003 19970 raft_consensus.cc:482] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:08:51.570113 19970 raft_consensus.cc:3037] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:51.574561 19970 raft_consensus.cc:504] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.575490 19970 leader_election.cc:288] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [CANDIDATE]: Term 1 election: Requested vote from peers d37a9d68b7ed4531870bb5079d60d7c1 (127.1.188.193:39063), 5969bcf9b8174d658cc25a4e06019ffa (127.1.188.194:34415)
I20260621 07:08:51.575893 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0a8b20865648ceaf3588b283bb78bf" candidate_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5969bcf9b8174d658cc25a4e06019ffa"
I20260621 07:08:51.575874 19638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0a8b20865648ceaf3588b283bb78bf" candidate_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37a9d68b7ed4531870bb5079d60d7c1"
I20260621 07:08:51.576092 19769 raft_consensus.cc:3037] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:51.576129 19638 raft_consensus.cc:3037] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:51.578389 19769 raft_consensus.cc:2454] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9cdc9c802ad4011877b39e3dfa1d663 in term 1.
I20260621 07:08:51.578822 19833 leader_election.cc:302] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5969bcf9b8174d658cc25a4e06019ffa, a9cdc9c802ad4011877b39e3dfa1d663; no voters:
I20260621 07:08:51.579090 19970 raft_consensus.cc:2781] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:51.579653 19970 raft_consensus.cc:686] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 1 LEADER]: Becoming Leader. State: Replica: a9cdc9c802ad4011877b39e3dfa1d663, State: Running, Role: LEADER
I20260621 07:08:51.580214 19970 consensus_queue.cc:237] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } }
I20260621 07:08:51.581365 19638 raft_consensus.cc:2454] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9cdc9c802ad4011877b39e3dfa1d663 in term 1.
I20260621 07:08:51.586390 19497 catalog_manager.cc:5495] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9cdc9c802ad4011877b39e3dfa1d663 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9cdc9c802ad4011877b39e3dfa1d663" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45299 } health_report { overall_health: HEALTHY } } }
I20260621 07:08:51.613796 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.606094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.1.188.195:45299)
0621 07:08:51.606280 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.606285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.606548 (+ 263us) server_negotiation.cc:407] Connection header received
0621 07:08:51.606756 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.606765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.606916 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.607105 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:51.607501 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.609009 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.610466 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.611436 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.611593 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.612149 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:51.613031 (+ 882us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:08:51.613211 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.613299 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.613411 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:08:51.624763 19769 raft_consensus.cc:1261] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 1 FOLLOWER]: Refusing update from remote peer a9cdc9c802ad4011877b39e3dfa1d663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:51.626039 19970 consensus_queue.cc:1035] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5969bcf9b8174d658cc25a4e06019ffa" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 34415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:51.626362 19638 raft_consensus.cc:1261] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 1 FOLLOWER]: Refusing update from remote peer a9cdc9c802ad4011877b39e3dfa1d663: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:51.627435 19970 consensus_queue.cc:1035] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37a9d68b7ed4531870bb5079d60d7c1" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:51.645047 19977 mvcc.cc:205] Tried to move back new op lower bound from 7299177396720058368 to 7299177396557471744. Current Snapshot: MvccSnapshot[applied={T|T < 7299177396720058368}]
I20260621 07:08:51.647805 19978 mvcc.cc:205] Tried to move back new op lower bound from 7299177396720058368 to 7299177396557471744. Current Snapshot: MvccSnapshot[applied={T|T < 7299177396720058368}]
I20260621 07:08:51.659668 19497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:51.659922 19497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:51.663501 19497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 618ed8cdeadb4c5891cc32d741b99806: Sending DeleteTablet for 3 replicas of tablet ae0a8b20865648ceaf3588b283bb78bf
I20260621 07:08:51.664904 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0a8b20865648ceaf3588b283bb78bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:51 UTC) from {username='slave'} at 127.0.0.1:41338
I20260621 07:08:51.665009 19880 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0a8b20865648ceaf3588b283bb78bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:51 UTC) from {username='slave'} at 127.0.0.1:49268
I20260621 07:08:51.664990 19618 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0a8b20865648ceaf3588b283bb78bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:51 UTC) from {username='slave'} at 127.0.0.1:46492
I20260621 07:08:51.667161 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19556
I20260621 07:08:51.677392 19988 tablet_replica.cc:330] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: stopping tablet replica
I20260621 07:08:51.678014 19988 raft_consensus.cc:2229] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:51.678747 19988 raft_consensus.cc:2258] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:51.681087 19988 ts_tablet_manager.cc:1905] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:51.688443 19988 ts_tablet_manager.cc:1918] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:51.688707 19988 log.cc:1196] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/ae0a8b20865648ceaf3588b283bb78bf
I20260621 07:08:51.689321 19988 ts_tablet_manager.cc:1939] T ae0a8b20865648ceaf3588b283bb78bf P d37a9d68b7ed4531870bb5079d60d7c1: Deleting consensus metadata
I20260621 07:08:51.691526 19484 catalog_manager.cc:4841] TS d37a9d68b7ed4531870bb5079d60d7c1 (127.1.188.193:39063): tablet ae0a8b20865648ceaf3588b283bb78bf (table test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]) successfully deleted
I20260621 07:08:51.698556 19987 tablet_replica.cc:330] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: stopping tablet replica
I20260621 07:08:51.699136 19987 raft_consensus.cc:2229] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:51.699865 19987 raft_consensus.cc:2258] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:51.707073 19987 ts_tablet_manager.cc:1905] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:51.712007 19986 tablet_replica.cc:330] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: stopping tablet replica
I20260621 07:08:51.712478 19986 raft_consensus.cc:2229] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:08:51.713227 19986 raft_consensus.cc:2258] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:51.714229 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.668825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.1.188.193:39063)
0621 07:08:51.669070 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.669078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.693386 (+ 24308us) server_negotiation.cc:407] Connection header received
0621 07:08:51.693961 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.693970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.694099 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.694199 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:51.695090 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.696716 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.702297 (+ 5581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.703192 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.703315 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.712019 (+ 8704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:51.712033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:51.712054 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:51.712889 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:51.712954 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:51.712985 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:51.713140 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.713723 (+ 583us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.713866 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 07:08:51.714923 19678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:51.715495 19986 ts_tablet_manager.cc:1905] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:51.717819 19987 ts_tablet_manager.cc:1918] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:51.718029 19987 log.cc:1196] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/ae0a8b20865648ceaf3588b283bb78bf
I20260621 07:08:51.718869 19987 ts_tablet_manager.cc:1939] T ae0a8b20865648ceaf3588b283bb78bf P 5969bcf9b8174d658cc25a4e06019ffa: Deleting consensus metadata
I20260621 07:08:51.721457 19481 catalog_manager.cc:4841] TS 5969bcf9b8174d658cc25a4e06019ffa (127.1.188.194:34415): tablet ae0a8b20865648ceaf3588b283bb78bf (table test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]) successfully deleted
I20260621 07:08:51.723124 19986 ts_tablet_manager.cc:1918] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:51.723317 19986 log.cc:1196] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/ae0a8b20865648ceaf3588b283bb78bf
I20260621 07:08:51.724112 19986 ts_tablet_manager.cc:1939] T ae0a8b20865648ceaf3588b283bb78bf P a9cdc9c802ad4011877b39e3dfa1d663: Deleting consensus metadata
I20260621 07:08:51.726416 19482 catalog_manager.cc:4841] TS a9cdc9c802ad4011877b39e3dfa1d663 (127.1.188.195:45299): tablet ae0a8b20865648ceaf3588b283bb78bf (table test-table [id=bfc7c966fe734331b2a710c3bcd7cb94]) successfully deleted
I20260621 07:08:51.812698 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19556
I20260621 07:08:51.831768 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19687
I20260621 07:08:51.839855 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.832651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.1.188.194:34415)
0621 07:08:51.832869 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.832876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.832898 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:08:51.832969 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.832974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.833108 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.833221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:51.833634 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.834832 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.836199 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:51.836869 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:51.837005 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:51.838066 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:51.838079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:51.838098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:51.838938 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:51.839005 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:51.839040 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:51.839161 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:51.839242 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 07:08:51.839364 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:08:51.840641 19809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:51.980350 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19687
I20260621 07:08:51.998055 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19818
I20260621 07:08:52.005566 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:51.998919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.1.188.195:45299)
0621 07:08:51.999096 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:51.999102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:51.999182 (+ 80us) server_negotiation.cc:407] Connection header received
0621 07:08:51.999456 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:51.999462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:51.999554 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:51.999691 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:52.000103 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:52.000896 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:52.002212 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:52.002663 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:52.002809 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:52.004103 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:52.004112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:52.004126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:52.004760 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:52.004810 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:52.004836 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:52.004959 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:52.005105 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 07:08:52.005208 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:08:52.006385 19940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:52.129403 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19818
I20260621 07:08:52.149693 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 19466
I20260621 07:08:52.150882 19527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:52.264866 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 19466
2026-06-21T07:08:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3317 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:08:52 dist-test-slave-hlq1 krb5kdc[19996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:08:52 dist-test-slave-hlq1 krb5kdc[19996](info): set up 2 sockets
Jun 21 07:08:52 dist-test-slave-hlq1 krb5kdc[19996](info): commencing operation
krb5kdc: starting...
W20260621 07:08:53.376034 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:08:53 dist-test-slave-hlq1 krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:08:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:08:53.664695 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33933
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:54.041112 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:54.041496 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:54.041666 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:54.049299 20009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:54.049458 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:54.049569 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:54.049678 20009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:54.049757 20009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:54.060705 20009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:33933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:54.062372 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:54.063972 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:54.076799 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:54.076743 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:54.077097 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:54.078105 20009 server_base.cc:1034] running on GCE node
I20260621 07:08:54.078943 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:54.080585 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:54.081802 20009 hybrid_clock.cc:648] HybridClock initialized: now 1782025734081756 us; error 33 us; skew 500 ppm
I20260621 07:08:54.082209 20009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:54 dist-test-slave-hlq1 krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025734, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:54.085762 20009 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:08:54.087477 20009 webserver.cc:460] Webserver started at http://127.1.188.254:44707/ using document root <none> and password file <none>
I20260621 07:08:54.088284 20009 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:54.088436 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:54.088785 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:54.092283 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9838a116827e4ebd9bac7665a25f2718"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "b2e070a79c4cf576f7100c6fa298ad30"
server_key_iv: "27650b5fb31e20552b5a8c5661693235"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.093109 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9838a116827e4ebd9bac7665a25f2718"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "b2e070a79c4cf576f7100c6fa298ad30"
server_key_iv: "27650b5fb31e20552b5a8c5661693235"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.098907 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:08:54.102818 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:54.104290 20009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:08:54.104480 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9838a116827e4ebd9bac7665a25f2718"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "b2e070a79c4cf576f7100c6fa298ad30"
server_key_iv: "27650b5fb31e20552b5a8c5661693235"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.104701 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:54.121649 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:54.130728 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:54.131170 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:54.154079 20009 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:33933
I20260621 07:08:54.154080 20075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:33933 every 8 connection(s)
I20260621 07:08:54.155974 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:54.160640 20076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:54.162809 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20009
I20260621 07:08:54.163059 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:08:54.163690 1779 external_mini_cluster.cc:1428] Setting key 98ca5a8db666df5cdd3a264588b2871a
I20260621 07:08:54.174181 20076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: Bootstrap starting.
I20260621 07:08:54.178864 20076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: Neither blocks nor log segments found. Creating new log.
Jun 21 07:08:54 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:54.180203 20076 log.cc:824] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:54.183389 20076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: No bootstrap required, opened a new log
I20260621 07:08:54.188973 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:54.165768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.1.188.254:33933)
0621 07:08:54.166652 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:54.166667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:54.166716 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:08:54.168649 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:54.168674 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:54.169097 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:54.169431 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:54.171393 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.172922 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.174856 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.175709 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.175847 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:54.181156 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:54.181165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:54.181184 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:54.181230 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:54.184578 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:54.185321 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:54.185328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:54.185334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:54.185505 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:54.185974 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:54.185982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:54.185984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:54.186560 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:54.186899 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:54.187344 (+ 445us) server_negotiation.cc:299] Negotiation successful
0621 07:08:54.187837 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":415,"threads_started":1}
I20260621 07:08:54.192643 20076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } }
I20260621 07:08:54.193058 20076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:54.193167 20076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9838a116827e4ebd9bac7665a25f2718, State: Initialized, Role: FOLLOWER
I20260621 07:08:54.193770 20076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } }
I20260621 07:08:54.194038 20076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:54.194190 20076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:54.194363 20076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:54.197345 20076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } }
I20260621 07:08:54.197882 20076 leader_election.cc:302] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9838a116827e4ebd9bac7665a25f2718; no voters:
I20260621 07:08:54.198534 20076 leader_election.cc:288] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:08:54.198737 20081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:54.199556 20081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [term 1 LEADER]: Becoming Leader. State: Replica: 9838a116827e4ebd9bac7665a25f2718, State: Running, Role: LEADER
I20260621 07:08:54.200156 20081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } }
I20260621 07:08:54.200767 20076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:54.203822 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9838a116827e4ebd9bac7665a25f2718. Latest consensus state: current_term: 1 leader_uuid: "9838a116827e4ebd9bac7665a25f2718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } } }
I20260621 07:08:54.204392 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:54.203795 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9838a116827e4ebd9bac7665a25f2718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9838a116827e4ebd9bac7665a25f2718" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33933 } } }
I20260621 07:08:54.206411 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718 [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:54.207144 20089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:54.213516 20089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:54.223320 20089 catalog_manager.cc:1347] Generated new cluster ID: a70648b97ec94c4292fe94e4ac2e3e01
I20260621 07:08:54.223475 20089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:54.238782 20089 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:08:54.240365 20089 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:54.252519 20089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: Generated new TSK 0
I20260621 07:08:54.253548 20089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:08:54.336804 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:54.722040 20102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:54.722303 20102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:54.722493 20102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:54.729547 20102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:54.729641 20102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:54.729840 20102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:54.739351 20102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:54.740693 20102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:54.742058 20102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:54.755491 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:54.755434 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:54.756042 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:54.756420 20102 server_base.cc:1034] running on GCE node
I20260621 07:08:54.756984 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:54.757894 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:54.759119 20102 hybrid_clock.cc:648] HybridClock initialized: now 1782025734759083 us; error 46 us; skew 500 ppm
I20260621 07:08:54.759487 20102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:54 dist-test-slave-hlq1 krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025734, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:54.765549 20102 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:08:54.767071 20102 webserver.cc:460] Webserver started at http://127.1.188.193:39853/ using document root <none> and password file <none>
I20260621 07:08:54.767882 20102 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:54.768028 20102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:54.768357 20102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:54.771272 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "3a1e4903efe64e72ab486bdad8fde778"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "ea19a21508d9b76e64fe1c1c958880af"
server_key_iv: "64d14b1089ca085148726365dc4dae36"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.772147 20102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "3a1e4903efe64e72ab486bdad8fde778"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "ea19a21508d9b76e64fe1c1c958880af"
server_key_iv: "64d14b1089ca085148726365dc4dae36"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.777223 20102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:08:54.780757 20117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:54.782043 20102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:08:54.782207 20102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "3a1e4903efe64e72ab486bdad8fde778"
format_stamp: "Formatted at 2026-06-21 07:08:54 on dist-test-slave-hlq1"
server_key: "ea19a21508d9b76e64fe1c1c958880af"
server_key_iv: "64d14b1089ca085148726365dc4dae36"
server_key_version: "encryptionkey@0"
I20260621 07:08:54.782409 20102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:54.796769 20102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:54.801049 20102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:54.801379 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:54.802415 20102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:54.803833 20102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:54.803932 20102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:54.804072 20102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:54.804147 20102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:54.838181 20102 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:45065
I20260621 07:08:54.838267 20229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:45065 every 8 connection(s)
I20260621 07:08:54.839596 20102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:54.841121 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20102
I20260621 07:08:54.841323 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:08:54.841909 1779 external_mini_cluster.cc:1428] Setting key c033883f22f39d444ed43636bfa2aa85
Jun 21 07:08:54 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025734, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:54.863250 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:54.842468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60473 (local address 127.1.188.254:33933)
0621 07:08:54.842969 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:54.842973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:54.847491 (+ 4518us) server_negotiation.cc:407] Connection header received
0621 07:08:54.847568 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:54.847576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:54.847676 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:54.847800 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:54.849368 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.850530 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.852470 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.852876 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.852981 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:54.857061 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:54.857066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:54.857070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:54.857102 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:54.859689 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:54.860502 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:54.860510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:54.860517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:54.860679 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:54.861131 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:54.861138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:54.861144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:54.861556 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:54.861726 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:54.862470 (+ 744us) server_negotiation.cc:299] Negotiation successful
0621 07:08:54.862738 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260621 07:08:54.863835 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:54.843063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33933 (local address 127.1.188.193:60473)
0621 07:08:54.843926 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:54.843981 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:54.845722 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:54.848026 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:54.848034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:54.848555 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:54.849143 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:54.849203 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.850704 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.850708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:54.852297 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:54.852314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:54.853109 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:54.853119 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:54.853452 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:54.854207 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:54.854241 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:54.856852 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:54.859847 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:54.859856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:54.859871 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:54.860351 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:54.860814 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:54.860818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:54.860820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:54.861003 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:54.861703 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:54.861709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:54.862026 (+ 317us) client_negotiation.cc:769] Sending connection context
0621 07:08:54.862292 (+ 266us) client_negotiation.cc:240] Negotiation successful
0621 07:08:54.862663 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260621 07:08:54.865581 20230 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33933
I20260621 07:08:54.865922 20230 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:54.866640 20230 heartbeater.cc:511] Master 127.1.188.254:33933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
I20260621 07:08:54.868956 20041 ts_manager.cc:194] Registered new tserver with Master: 3a1e4903efe64e72ab486bdad8fde778 (127.1.188.193:45065)
I20260621 07:08:54.870623 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:60473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:08:54.896582 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:55.230155 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:55.230492 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:55.230667 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:55.237923 20236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:55.238078 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:55.238265 20236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:55.247995 20236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:55.249460 20236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:55.250736 20236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:55.263458 20241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:55.263535 20244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:55.263490 20242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:55.264354 20236 server_base.cc:1034] running on GCE node
I20260621 07:08:55.264981 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:55.265978 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:55.267253 20236 hybrid_clock.cc:648] HybridClock initialized: now 1782025735267196 us; error 72 us; skew 500 ppm
I20260621 07:08:55.267717 20236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:55 dist-test-slave-hlq1 krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025735, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:55.270974 20236 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:08:55.272727 20236 webserver.cc:460] Webserver started at http://127.1.188.194:45073/ using document root <none> and password file <none>
I20260621 07:08:55.273458 20236 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:55.273613 20236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:55.273936 20236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:55.276939 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "743193d25ddd4c1db33173a5133597d2"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "0de7d9a3e4a10e1a52e660412d483140"
server_key_iv: "64dcee686228cfe3e8c3f9d1caa3bc32"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.277750 20236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "743193d25ddd4c1db33173a5133597d2"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "0de7d9a3e4a10e1a52e660412d483140"
server_key_iv: "64dcee686228cfe3e8c3f9d1caa3bc32"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.282897 20236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 07:08:55.286567 20251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.288025 20236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:08:55.288210 20236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "743193d25ddd4c1db33173a5133597d2"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "0de7d9a3e4a10e1a52e660412d483140"
server_key_iv: "64dcee686228cfe3e8c3f9d1caa3bc32"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.288429 20236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:55.310940 20236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:55.314970 20236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:55.315268 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:55.316426 20236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:55.317925 20236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:55.318035 20236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.318192 20236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:55.318264 20236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.351280 20236 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:41587
I20260621 07:08:55.351361 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:41587 every 8 connection(s)
I20260621 07:08:55.352568 20236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:55.360874 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20236
I20260621 07:08:55.361065 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:08:55.361584 1779 external_mini_cluster.cc:1428] Setting key 27cdf389ce8b243078cc4a6b07621b6a
Jun 21 07:08:55 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025735, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:55.374007 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.355166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:34663 (local address 127.1.188.254:33933)
0621 07:08:55.355374 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.355378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.356486 (+ 1108us) server_negotiation.cc:407] Connection header received
0621 07:08:55.358329 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.358334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.358422 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.358528 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:55.360011 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.360947 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.362682 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.363137 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.363308 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.367080 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:55.367089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:55.367097 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:55.367143 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:55.370775 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.371555 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.371564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.371570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.371739 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.372156 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.372163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.372170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.372530 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:55.372650 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.373547 (+ 897us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.373754 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 07:08:55.375458 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.355641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33933 (local address 127.1.188.194:34663)
0621 07:08:55.356326 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:55.356370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:55.358051 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:55.358743 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:55.358751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:55.359250 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:55.359846 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.359863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.361148 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.361156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.362539 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.362554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.363487 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.363495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.363741 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.364496 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:55.364532 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:55.366831 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:55.370927 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:55.370933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:55.370945 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:55.371359 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:55.371847 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:55.371850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:55.371853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:55.372036 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:55.372661 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:55.372671 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:55.373133 (+ 462us) client_negotiation.cc:769] Sending connection context
0621 07:08:55.373454 (+ 321us) client_negotiation.cc:240] Negotiation successful
0621 07:08:55.374029 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":251,"threads_started":1}
I20260621 07:08:55.376880 20364 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33933
I20260621 07:08:55.377223 20364 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:55.377909 20364 heartbeater.cc:511] Master 127.1.188.254:33933 requested a full tablet report, sending...
I20260621 07:08:55.379321 20041 ts_manager.cc:194] Registered new tserver with Master: 743193d25ddd4c1db33173a5133597d2 (127.1.188.194:41587)
I20260621 07:08:55.380175 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:34663
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:08:55.418457 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:33021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:08:55.746920 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:55.747256 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:55.747488 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:55.754827 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:55.754979 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:55.755179 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:55.765067 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:33021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:55.766439 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:55.767728 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:55.780692 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:55.780887 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:55.781195 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:55.781649 20369 server_base.cc:1034] running on GCE node
I20260621 07:08:55.782249 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:55.783172 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:55.784403 20369 hybrid_clock.cc:648] HybridClock initialized: now 1782025735784343 us; error 52 us; skew 500 ppm
I20260621 07:08:55.784718 20369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:08:55 dist-test-slave-hlq1 krb5kdc[19996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025735, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:08:55.788040 20369 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:08:55.789495 20369 webserver.cc:460] Webserver started at http://127.1.188.195:45757/ using document root <none> and password file <none>
I20260621 07:08:55.790215 20369 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:55.790360 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:55.790668 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:55.793646 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "6161ab8a4e694a6199e60a02cf029d32"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "2bb929b5181ebc5e7744095b3f785d1a"
server_key_iv: "f0c2298c87b36cccacec14fe97ff9e76"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.794433 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "6161ab8a4e694a6199e60a02cf029d32"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "2bb929b5181ebc5e7744095b3f785d1a"
server_key_iv: "f0c2298c87b36cccacec14fe97ff9e76"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.799458 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260621 07:08:55.802829 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.804180 20369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:08:55.804355 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "6161ab8a4e694a6199e60a02cf029d32"
format_stamp: "Formatted at 2026-06-21 07:08:55 on dist-test-slave-hlq1"
server_key: "2bb929b5181ebc5e7744095b3f785d1a"
server_key_iv: "f0c2298c87b36cccacec14fe97ff9e76"
server_key_version: "encryptionkey@0"
I20260621 07:08:55.804551 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:55.831359 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:55.835486 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:55.835777 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:55.836726 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:55.838312 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:55.838413 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.838526 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:55.838594 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:55.872022 20369 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:38219
I20260621 07:08:55.872105 20496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:38219 every 8 connection(s)
I20260621 07:08:55.873618 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:55.873592 20230 heartbeater.cc:503] Master 127.1.188.254:33933 was elected leader, sending a full tablet report...
I20260621 07:08:55.881851 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20369
I20260621 07:08:55.882058 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:08:55.882661 1779 external_mini_cluster.cc:1428] Setting key 0193039f323496745d6e237115527730
Jun 21 07:08:55 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025735, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:08:55.897308 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.876160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:47513 (local address 127.1.188.254:33933)
0621 07:08:55.882343 (+ 6183us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.882350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.882370 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:08:55.882444 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.882448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.882534 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.882647 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:55.884824 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.885859 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.887602 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.888002 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.888092 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.891643 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:55.891647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:55.891650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:55.891681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:55.894172 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.894821 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.894825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.894827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.894941 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.895295 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.895298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.895300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.895628 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:55.895790 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.896705 (+ 915us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.896948 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6047,"thread_start_us":5933,"threads_started":1}
I20260621 07:08:55.898324 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.876738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33933 (local address 127.1.188.195:47513)
0621 07:08:55.877528 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:55.877581 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:55.879267 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:55.882922 (+ 3655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:55.882931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:55.883794 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:55.884633 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.884661 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.886016 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.886020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.887449 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.887469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.888229 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.888236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.888426 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.889132 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:08:55.889154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:55.891378 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:55.894295 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:55.894302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:55.894313 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:55.894716 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:55.895026 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:08:55.895029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:08:55.895031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:08:55.895201 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:08:55.895800 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:55.895806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:55.896232 (+ 426us) client_negotiation.cc:769] Sending connection context
0621 07:08:55.896554 (+ 322us) client_negotiation.cc:240] Negotiation successful
0621 07:08:55.897041 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":461,"threads_started":1}
I20260621 07:08:55.899631 20497 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33933
I20260621 07:08:55.899932 20497 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:55.900626 20497 heartbeater.cc:511] Master 127.1.188.254:33933 requested a full tablet report, sending...
I20260621 07:08:55.902065 20041 ts_manager.cc:194] Registered new tserver with Master: 6161ab8a4e694a6199e60a02cf029d32 (127.1.188.195:38219)
I20260621 07:08:55.902877 20041 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:47513
I20260621 07:08:55.909909 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:08:55.927109 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.914148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.1.188.254:33933)
0621 07:08:55.914310 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.914315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.914784 (+ 469us) server_negotiation.cc:407] Connection header received
0621 07:08:55.914918 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.914922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.914985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.915065 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:55.916524 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.917273 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.918569 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.918980 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.919080 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.921682 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:55.921687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:55.921690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:55.921733 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:55.924330 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.925146 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.925150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.925153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.925293 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:55.925723 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:55.925726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:55.925729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:55.926051 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:55.926266 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.926562 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.926722 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:08:55.931458 20041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:08:55.938546 20041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:08:55.962018 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.951760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40222)
0621 07:08:55.952845 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:55.952894 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:55.953088 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:55.953815 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:55.953825 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:55.953856 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:55.954369 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.954383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.956660 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.956666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.959203 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.959217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.960965 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.960971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.961158 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.961211 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 07:08:55.961355 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 07:08:55.961550 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":108,"thread_start_us":688,"threads_started":1}
I20260621 07:08:55.962584 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.953146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:38219 (local address 127.0.0.1:44558)
0621 07:08:55.953667 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:55.953686 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:55.953795 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:55.955600 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:55.955604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:55.955629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:55.956055 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.956068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.957749 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.957754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.959923 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.959934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.961524 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.961531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.961715 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.961755 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 07:08:55.961826 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:08:55.961960 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":37,"thread_start_us":239,"threads_started":1}
I20260621 07:08:55.962940 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.953495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.1.188.195:38219)
0621 07:08:55.955005 (+ 1510us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.955022 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.955054 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:08:55.955148 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.955155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.955362 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.956246 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:55.956514 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.957555 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.960068 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.961299 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.962118 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.962341 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.962409 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.962580 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":1137,"threads_started":1}
I20260621 07:08:55.963287 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.953704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:41587 (local address 127.0.0.1:50670)
0621 07:08:55.954301 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:55.954321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:55.954431 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:55.955812 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:55.955816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:55.955843 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:55.956219 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.956228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.958045 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.958054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.960419 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:55.960432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.962471 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.962477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:55.962649 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.962703 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 07:08:55.962793 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 07:08:55.962912 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":30,"thread_start_us":742,"threads_started":1}
I20260621 07:08:55.963297 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.952126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.1.188.193:45065)
0621 07:08:55.952806 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.952814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.952991 (+ 177us) server_negotiation.cc:407] Connection header received
0621 07:08:55.953298 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.953309 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.953578 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.954590 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:55.954871 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.956512 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.959378 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.960784 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.960938 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.961175 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.962779 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.962968 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":346,"threads_started":1}
I20260621 07:08:55.963956 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:55.954250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.1.188.194:41587)
0621 07:08:55.955236 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:55.955244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:55.955270 (+ 26us) server_negotiation.cc:407] Connection header received
0621 07:08:55.955366 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:55.955372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:55.955610 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:55.955828 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:55.956391 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.957887 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.960591 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:55.962019 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:55.962139 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:55.962316 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:55.963542 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0621 07:08:55.963723 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":841,"threads_started":1}
I20260621 07:08:55.966925 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b08df34d448b49ebb476237a54889 (DEFAULT_TABLE table=test-table [id=dea6dea6808d4084be3a3d5b568a79d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:55.966881 20165 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b08df34d448b49ebb476237a54889 (DEFAULT_TABLE table=test-table [id=dea6dea6808d4084be3a3d5b568a79d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:55.967545 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b08df34d448b49ebb476237a54889 (DEFAULT_TABLE table=test-table [id=dea6dea6808d4084be3a3d5b568a79d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:55.968389 20165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b08df34d448b49ebb476237a54889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:55.968413 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b08df34d448b49ebb476237a54889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:55.968734 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b08df34d448b49ebb476237a54889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:55.980088 20516 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Bootstrap starting.
I20260621 07:08:55.982455 20517 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Bootstrap starting.
I20260621 07:08:55.983222 20518 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Bootstrap starting.
I20260621 07:08:55.983810 20516 tablet_bootstrap.cc:654] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:55.985179 20516 log.cc:824] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:55.986899 20517 tablet_bootstrap.cc:654] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:55.987136 20518 tablet_bootstrap.cc:654] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:55.988191 20517 log.cc:824] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:55.988304 20518 log.cc:824] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:55.990357 20516 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: No bootstrap required, opened a new log
I20260621 07:08:55.990658 20516 ts_tablet_manager.cc:1397] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260621 07:08:55.991125 20518 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: No bootstrap required, opened a new log
I20260621 07:08:55.991214 20517 tablet_bootstrap.cc:492] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: No bootstrap required, opened a new log
I20260621 07:08:55.991452 20518 ts_tablet_manager.cc:1397] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 07:08:55.991529 20517 ts_tablet_manager.cc:1397] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 07:08:55.999092 20518 raft_consensus.cc:348] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:55.999521 20518 raft_consensus.cc:374] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:55.999619 20518 raft_consensus.cc:729] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6161ab8a4e694a6199e60a02cf029d32, State: Initialized, Role: FOLLOWER
I20260621 07:08:56.000209 20518 consensus_queue.cc:260] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.001400 20497 heartbeater.cc:503] Master 127.1.188.254:33933 was elected leader, sending a full tablet report...
I20260621 07:08:56.002101 20518 ts_tablet_manager.cc:1428] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260621 07:08:56.001849 20516 raft_consensus.cc:348] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.002358 20516 raft_consensus.cc:374] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:56.002470 20516 raft_consensus.cc:729] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1e4903efe64e72ab486bdad8fde778, State: Initialized, Role: FOLLOWER
I20260621 07:08:56.003073 20516 consensus_queue.cc:260] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.003847 20517 raft_consensus.cc:348] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.004269 20517 raft_consensus.cc:374] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:56.004361 20517 raft_consensus.cc:729] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743193d25ddd4c1db33173a5133597d2, State: Initialized, Role: FOLLOWER
I20260621 07:08:56.004926 20516 ts_tablet_manager.cc:1428] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260621 07:08:56.004961 20517 consensus_queue.cc:260] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.006662 20522 raft_consensus.cc:482] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:56.006836 20364 heartbeater.cc:503] Master 127.1.188.254:33933 was elected leader, sending a full tablet report...
I20260621 07:08:56.006991 20522 raft_consensus.cc:504] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.008256 20517 ts_tablet_manager.cc:1428] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260621 07:08:56.011026 20522 leader_election.cc:288] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a1e4903efe64e72ab486bdad8fde778 (127.1.188.193:45065), 743193d25ddd4c1db33173a5133597d2 (127.1.188.194:41587)
I20260621 07:08:56.016489 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.010482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.1.188.195:58711)
0621 07:08:56.010767 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.010787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:56.010903 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:56.011544 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:56.011552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:56.011599 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:56.011847 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:56.011862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.013338 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.013342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:56.014586 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:56.014601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.015756 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.015764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:56.015956 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.015981 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:08:56.016088 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 07:08:56.016220 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260621 07:08:56.016489 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.010567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:58711 (local address 127.1.188.193:45065)
0621 07:08:56.010765 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.010771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.011041 (+ 270us) server_negotiation.cc:407] Connection header received
0621 07:08:56.011114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.011121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.011234 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.011326 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:56.012050 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.013104 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.014740 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.015611 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.015774 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.015848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.016112 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.016234 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:08:56.017405 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b08df34d448b49ebb476237a54889" candidate_uuid: "6161ab8a4e694a6199e60a02cf029d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1e4903efe64e72ab486bdad8fde778" is_pre_election: true
I20260621 07:08:56.017953 20185 raft_consensus.cc:2454] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6161ab8a4e694a6199e60a02cf029d32 in term 0.
I20260621 07:08:56.018118 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.011602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:53037 (local address 127.1.188.194:41587)
0621 07:08:56.011785 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.011791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.012627 (+ 836us) server_negotiation.cc:407] Connection header received
0621 07:08:56.012729 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.012734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.012798 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.012891 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:56.013503 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.014187 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.015884 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.017165 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.017349 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.017425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.017744 (+ 319us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.017877 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:08:56.018152 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.011239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:41587 (local address 127.1.188.195:53037)
0621 07:08:56.012525 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.012540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:56.012639 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:56.012916 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:56.012919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:56.012938 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:56.013334 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:56.013343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.014336 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.014341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:56.015741 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:56.015752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.017362 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.017368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:56.017549 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.017575 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 07:08:56.017690 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 07:08:56.017823 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":978,"threads_started":1}
I20260621 07:08:56.018661 20388 leader_election.cc:302] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a1e4903efe64e72ab486bdad8fde778, 6161ab8a4e694a6199e60a02cf029d32; no voters:
I20260621 07:08:56.019061 20522 raft_consensus.cc:2781] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:08:56.019227 20522 raft_consensus.cc:482] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:08:56.019343 20522 raft_consensus.cc:3037] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:56.019251 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b08df34d448b49ebb476237a54889" candidate_uuid: "6161ab8a4e694a6199e60a02cf029d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743193d25ddd4c1db33173a5133597d2" is_pre_election: true
I20260621 07:08:56.019798 20319 raft_consensus.cc:2454] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6161ab8a4e694a6199e60a02cf029d32 in term 0.
I20260621 07:08:56.022471 20522 raft_consensus.cc:504] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.023123 20522 leader_election.cc:288] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [CANDIDATE]: Term 1 election: Requested vote from peers 3a1e4903efe64e72ab486bdad8fde778 (127.1.188.193:45065), 743193d25ddd4c1db33173a5133597d2 (127.1.188.194:41587)
I20260621 07:08:56.023582 20185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b08df34d448b49ebb476237a54889" candidate_uuid: "6161ab8a4e694a6199e60a02cf029d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1e4903efe64e72ab486bdad8fde778"
I20260621 07:08:56.023682 20319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b08df34d448b49ebb476237a54889" candidate_uuid: "6161ab8a4e694a6199e60a02cf029d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743193d25ddd4c1db33173a5133597d2"
I20260621 07:08:56.023830 20185 raft_consensus.cc:3037] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:56.023871 20319 raft_consensus.cc:3037] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:56.027103 20319 raft_consensus.cc:2454] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6161ab8a4e694a6199e60a02cf029d32 in term 1.
I20260621 07:08:56.027103 20185 raft_consensus.cc:2454] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6161ab8a4e694a6199e60a02cf029d32 in term 1.
I20260621 07:08:56.027601 20388 leader_election.cc:302] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a1e4903efe64e72ab486bdad8fde778, 6161ab8a4e694a6199e60a02cf029d32; no voters:
I20260621 07:08:56.027892 20522 raft_consensus.cc:2781] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:56.028354 20522 raft_consensus.cc:686] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 1 LEADER]: Becoming Leader. State: Replica: 6161ab8a4e694a6199e60a02cf029d32, State: Running, Role: LEADER
I20260621 07:08:56.028878 20522 consensus_queue.cc:237] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } }
I20260621 07:08:56.033288 20041 catalog_manager.cc:5495] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6161ab8a4e694a6199e60a02cf029d32 (127.1.188.195). New cstate: current_term: 1 leader_uuid: "6161ab8a4e694a6199e60a02cf029d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6161ab8a4e694a6199e60a02cf029d32" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 38219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 } health_report { overall_health: UNKNOWN } } }
I20260621 07:08:56.050413 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.044978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.1.188.195:38219)
0621 07:08:56.045200 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.045208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.045282 (+ 74us) server_negotiation.cc:407] Connection header received
0621 07:08:56.045360 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.045363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.045453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.045604 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:56.046008 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.046722 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.048137 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.048558 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.048690 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.049307 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:56.049795 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:08:56.049974 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.050131 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.050179 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:08:56.058714 20319 raft_consensus.cc:1261] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 1 FOLLOWER]: Refusing update from remote peer 6161ab8a4e694a6199e60a02cf029d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:56.058703 20185 raft_consensus.cc:1261] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 1 FOLLOWER]: Refusing update from remote peer 6161ab8a4e694a6199e60a02cf029d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:56.059762 20526 consensus_queue.cc:1035] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1e4903efe64e72ab486bdad8fde778" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:56.060133 20522 consensus_queue.cc:1035] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "743193d25ddd4c1db33173a5133597d2" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 41587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:56.073627 20531 mvcc.cc:205] Tried to move back new op lower bound from 7299177414885392384 to 7299177414778204160. Current Snapshot: MvccSnapshot[applied={T|T < 7299177414885392384 or (T in {7299177414885392384})}]
I20260621 07:08:56.088312 20041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:56.088541 20041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:56.092315 20041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9838a116827e4ebd9bac7665a25f2718: Sending DeleteTablet for 3 replicas of tablet 5d5b08df34d448b49ebb476237a54889
I20260621 07:08:56.093604 20432 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b08df34d448b49ebb476237a54889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:56 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:44558
I20260621 07:08:56.093853 20165 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b08df34d448b49ebb476237a54889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:56 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:40222
I20260621 07:08:56.094089 20299 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b08df34d448b49ebb476237a54889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:56 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:50670
I20260621 07:08:56.096429 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20102
I20260621 07:08:56.109743 20541 tablet_replica.cc:330] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: stopping tablet replica
I20260621 07:08:56.110176 20541 raft_consensus.cc:2229] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:56.110700 20541 raft_consensus.cc:2258] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:56.113374 20541 ts_tablet_manager.cc:1905] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 07:08:56 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:08:56.121958 20539 tablet_replica.cc:330] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: stopping tablet replica
I20260621 07:08:56.122462 20539 raft_consensus.cc:2229] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:08:56.123165 20539 raft_consensus.cc:2258] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:56.123996 20540 tablet_replica.cc:330] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: stopping tablet replica
I20260621 07:08:56.124145 20541 ts_tablet_manager.cc:1918] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:56.124346 20541 log.cc:1196] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/5d5b08df34d448b49ebb476237a54889
I20260621 07:08:56.124426 20540 raft_consensus.cc:2229] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:56.124856 20541 ts_tablet_manager.cc:1939] T 5d5b08df34d448b49ebb476237a54889 P 743193d25ddd4c1db33173a5133597d2: Deleting consensus metadata
I20260621 07:08:56.124959 20540 raft_consensus.cc:2258] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:56.126674 20028 catalog_manager.cc:4841] TS 743193d25ddd4c1db33173a5133597d2 (127.1.188.194:41587): tablet 5d5b08df34d448b49ebb476237a54889 (table test-table [id=dea6dea6808d4084be3a3d5b568a79d9]) successfully deleted
I20260621 07:08:56.128572 20539 ts_tablet_manager.cc:1905] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:56.133510 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.098272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.1.188.193:45065)
0621 07:08:56.098508 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.098517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.102439 (+ 3922us) server_negotiation.cc:407] Connection header received
0621 07:08:56.103025 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.103031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.103117 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.103201 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:56.106655 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.107915 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.113639 (+ 5724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.114373 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.114478 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.126669 (+ 12191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:56.126688 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:56.126707 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:56.126760 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:56.129921 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.130720 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.130727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.130730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.130860 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.131280 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.131284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.131286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.132181 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:56.132528 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.132872 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.133117 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:08:56.134543 20540 ts_tablet_manager.cc:1905] T 5d5b08df34d448b49ebb476237a54889 P 3a1e4903efe64e72ab486bdad8fde778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:56.135062 20225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:56.136278 20539 ts_tablet_manager.cc:1918] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:56.136477 20539 log.cc:1196] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/5d5b08df34d448b49ebb476237a54889
I20260621 07:08:56.137060 20539 ts_tablet_manager.cc:1939] T 5d5b08df34d448b49ebb476237a54889 P 6161ab8a4e694a6199e60a02cf029d32: Deleting consensus metadata
I20260621 07:08:56.139140 20028 catalog_manager.cc:4841] TS 6161ab8a4e694a6199e60a02cf029d32 (127.1.188.195:38219): tablet 5d5b08df34d448b49ebb476237a54889 (table test-table [id=dea6dea6808d4084be3a3d5b568a79d9]) successfully deleted
I20260621 07:08:56.226675 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20102
W20260621 07:08:56.245052 20028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.188.193:45065 (error 108)
I20260621 07:08:56.245283 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20236
I20260621 07:08:56.247243 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.246759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40242)
0621 07:08:56.246951 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.247066 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260621 07:08:56.247543 20028 catalog_manager.cc:4584] TS 3a1e4903efe64e72ab486bdad8fde778 (127.1.188.193:45065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d5b08df34d448b49ebb476237a54889: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Jun 21 07:08:56 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:08:56.261816 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.246422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.1.188.194:41587)
0621 07:08:56.246616 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.246622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.246641 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:08:56.246723 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.246729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.246826 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.246931 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:56.248419 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.249169 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.250477 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.250863 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.250998 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.255735 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:56.255758 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:56.255777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:56.255815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:56.258786 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.259745 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.259755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.259759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.259917 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.260366 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.260370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.260372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.260743 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:56.260971 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.261319 (+ 348us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.261488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:08:56.262532 20359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:56.301297 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.300726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40246)
0621 07:08:56.300974 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.301098 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 07:08:56.302487 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.302067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40260)
0621 07:08:56.302260 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.302347 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 07:08:56.353048 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20236
I20260621 07:08:56.363570 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.362927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40276)
0621 07:08:56.363192 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.363307 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 07:08:56.365003 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.364517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40288)
0621 07:08:56.364743 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.364841 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 07:08:56.371385 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20369
Jun 21 07:08:56 dist-test-slave-hlq1 krb5kdc[19996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:08:56.391360 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.372353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.1.188.195:38219)
0621 07:08:56.372629 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:56.372633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:56.372658 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:08:56.372877 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:56.372882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:56.372954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:56.373046 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:56.374937 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.376013 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.377875 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:56.378324 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:56.378414 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:56.384033 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:56.384050 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:56.384070 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:08:56.384118 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:56.388393 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.389092 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.389102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.389107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.389321 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:08:56.389678 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:08:56.389684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:08:56.389688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:08:56.390196 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:56.390512 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:56.390688 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 07:08:56.390933 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:08:56.392161 20492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:56.445916 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.445412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40292)
0621 07:08:56.445645 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.445743 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 07:08:56.447026 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.446606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40300)
0621 07:08:56.446855 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.446929 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260621 07:08:56.518154 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20369
I20260621 07:08:56.537500 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20009
I20260621 07:08:56.538645 20071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:56.669705 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:56.668906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:45065 (local address 127.0.0.1:40310)
0621 07:08:56.669290 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:56.669420 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260621 07:08:56.669701 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20009
2026-06-21T07:08:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T07:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:08:56Z Disabled control of system clock
I20260621 07:08:56.761766 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33613
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:08:57.134240 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:57.134598 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:57.134768 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:57.142448 20551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:08:57.142572 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:57.142669 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:57.142783 20551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:08:57.142866 20551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:08:57.152607 20551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:33613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:57.154208 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:57.156374 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:57.170012 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:57.169979 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:57.169938 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:57.171236 20551 server_base.cc:1034] running on GCE node
I20260621 07:08:57.172185 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:57.173786 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:57.175031 20551 hybrid_clock.cc:648] HybridClock initialized: now 1782025737174969 us; error 68 us; skew 500 ppm
I20260621 07:08:57.175493 20551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:57.178087 20551 webserver.cc:460] Webserver started at http://127.1.188.254:35449/ using document root <none> and password file <none>
I20260621 07:08:57.178871 20551 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:57.179009 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:57.179363 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:57.182631 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "103c169eb7e247678392c2bc5b628c0f"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "85820bcad74d19fd54f711422bce32a4"
server_key_iv: "800d978c9f38ca5fc101cdc645cee31b"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.183571 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "103c169eb7e247678392c2bc5b628c0f"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "85820bcad74d19fd54f711422bce32a4"
server_key_iv: "800d978c9f38ca5fc101cdc645cee31b"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.189308 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260621 07:08:57.193449 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:57.195022 20551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:08:57.195236 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "103c169eb7e247678392c2bc5b628c0f"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "85820bcad74d19fd54f711422bce32a4"
server_key_iv: "800d978c9f38ca5fc101cdc645cee31b"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.195515 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:57.221393 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:57.222481 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:57.222805 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:57.244026 20551 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:33613
I20260621 07:08:57.244055 20616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:33613 every 8 connection(s)
I20260621 07:08:57.245673 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:08:57.249509 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20551
I20260621 07:08:57.249734 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:08:57.250417 1779 external_mini_cluster.cc:1428] Setting key afa821e0fd6733d77edd3b6801e4188e
I20260621 07:08:57.250738 20617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:57.266415 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:57.252724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.1.188.254:33613)
0621 07:08:57.253842 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:57.253857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:57.253907 (+ 50us) server_negotiation.cc:407] Connection header received
0621 07:08:57.256111 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:57.256136 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:57.256661 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:57.257038 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:57.257682 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.259107 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.261216 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.261991 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.262147 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:57.263626 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:57.263634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:57.263652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:57.264337 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:57.264436 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:57.264461 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:57.264553 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:57.264798 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:08:57.265220 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":450,"threads_started":1}
I20260621 07:08:57.266847 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: Bootstrap starting.
I20260621 07:08:57.270637 20617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:57.271890 20617 log.cc:824] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:57.275040 20617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: No bootstrap required, opened a new log
I20260621 07:08:57.283021 20617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } }
I20260621 07:08:57.283514 20617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:57.283653 20617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103c169eb7e247678392c2bc5b628c0f, State: Initialized, Role: FOLLOWER
I20260621 07:08:57.284322 20617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } }
I20260621 07:08:57.284631 20617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:08:57.284792 20617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:08:57.285032 20617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:57.287516 20617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } }
I20260621 07:08:57.288103 20617 leader_election.cc:302] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 103c169eb7e247678392c2bc5b628c0f; no voters:
I20260621 07:08:57.288797 20617 leader_election.cc:288] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:08:57.289004 20622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:08:57.289784 20622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [term 1 LEADER]: Becoming Leader. State: Replica: 103c169eb7e247678392c2bc5b628c0f, State: Running, Role: LEADER
I20260621 07:08:57.290393 20622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } }
I20260621 07:08:57.291252 20617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:08:57.294047 20624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103c169eb7e247678392c2bc5b628c0f. Latest consensus state: current_term: 1 leader_uuid: "103c169eb7e247678392c2bc5b628c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } } }
I20260621 07:08:57.294389 20624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:57.293642 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "103c169eb7e247678392c2bc5b628c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103c169eb7e247678392c2bc5b628c0f" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 33613 } } }
I20260621 07:08:57.295228 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f [sys.catalog]: This master's current role is: LEADER
I20260621 07:08:57.296612 20627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:08:57.302889 20627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:08:57.313004 20627 catalog_manager.cc:1347] Generated new cluster ID: 3590a8bf47dd47e8a1ad2523e271b0c4
I20260621 07:08:57.313143 20627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:08:57.322918 20627 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:08:57.324661 20627 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:08:57.339558 20627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: Generated new TSK 0
I20260621 07:08:57.340826 20627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:08:57.354341 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--builtin_ntp_servers=127.1.188.212:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:08:57.736459 20641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:57.736800 20641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:57.736970 20641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:57.744469 20641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:57.744614 20641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:57.744796 20641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:08:57.754308 20641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:57.755774 20641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:57.757632 20641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:57.772079 20647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:57.772212 20649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:57.772066 20646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:57.773228 20641 server_base.cc:1034] running on GCE node
I20260621 07:08:57.773902 20641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:57.774856 20641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:57.776086 20641 hybrid_clock.cc:648] HybridClock initialized: now 1782025737776029 us; error 51 us; skew 500 ppm
I20260621 07:08:57.776430 20641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:57.783365 20641 webserver.cc:460] Webserver started at http://127.1.188.193:46451/ using document root <none> and password file <none>
I20260621 07:08:57.784196 20641 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:57.784330 20641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:57.784674 20641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:57.787731 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "5e643052591a45f4802cdb35ca9fd599"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "2ece202f86d07e6d49b63827d35d25e9"
server_key_iv: "abe2dfad9e328502741d8963c74b70a5"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.788573 20641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "5e643052591a45f4802cdb35ca9fd599"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "2ece202f86d07e6d49b63827d35d25e9"
server_key_iv: "abe2dfad9e328502741d8963c74b70a5"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.794056 20641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:08:57.798077 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:57.799757 20641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:08:57.799932 20641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "5e643052591a45f4802cdb35ca9fd599"
format_stamp: "Formatted at 2026-06-21 07:08:57 on dist-test-slave-hlq1"
server_key: "2ece202f86d07e6d49b63827d35d25e9"
server_key_iv: "abe2dfad9e328502741d8963c74b70a5"
server_key_version: "encryptionkey@0"
I20260621 07:08:57.800150 20641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:57.820899 20641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:57.821748 20641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:57.822065 20641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:57.823033 20641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:57.824592 20641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:57.824715 20641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:57.824851 20641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:57.824940 20641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:57.858029 20641 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:39467
I20260621 07:08:57.858116 20767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:39467 every 8 connection(s)
I20260621 07:08:57.859339 20641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:08:57.860445 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20641
I20260621 07:08:57.860631 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:08:57.861166 1779 external_mini_cluster.cc:1428] Setting key 04e40a05acfa5447639c120df9770fc3
I20260621 07:08:57.864697 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--builtin_ntp_servers=127.1.188.212:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:08:57.872601 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:57.862132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60493 (local address 127.1.188.254:33613)
0621 07:08:57.862549 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:57.862554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:57.864020 (+ 1466us) server_negotiation.cc:407] Connection header received
0621 07:08:57.865895 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:57.865903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:57.866001 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:57.866117 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:57.867157 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.868072 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.869602 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.870014 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.870194 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:57.871711 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:57.871717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:57.871722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:57.871749 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:57.871787 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:57.871800 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:57.871893 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:57.872128 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:08:57.872240 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260621 07:08:57.873571 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:57.862784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33613 (local address 127.1.188.193:60493)
0621 07:08:57.863885 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:57.863920 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:57.865633 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:57.866338 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:57.866346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:57.866431 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:57.866995 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:57.867011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.868226 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.868231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:57.869455 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:57.869466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:57.870274 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:57.870281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:57.870522 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:57.871351 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:57.871353 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:57.871391 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:57.871520 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:57.871902 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:57.871915 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:57.871923 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:08:57.872101 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 07:08:57.872489 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
I20260621 07:08:57.875156 20768 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33613
I20260621 07:08:57.875561 20768 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:57.876323 20768 heartbeater.cc:511] Master 127.1.188.254:33613 requested a full tablet report, sending...
I20260621 07:08:57.878588 20582 ts_manager.cc:194] Registered new tserver with Master: 5e643052591a45f4802cdb35ca9fd599 (127.1.188.193:39467)
I20260621 07:08:57.880802 20582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.193:60493
W20260621 07:08:58.217546 20772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:58.217816 20772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:58.217900 20772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:58.225075 20772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:58.225170 20772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:58.225277 20772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:08:58.234525 20772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:58.235848 20772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:58.237495 20772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:58.250329 20778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:58.250357 20777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:58.252480 20780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:58.257038 20772 server_base.cc:1034] running on GCE node
I20260621 07:08:58.257648 20772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:58.258704 20772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:58.259963 20772 hybrid_clock.cc:648] HybridClock initialized: now 1782025738259906 us; error 42 us; skew 500 ppm
I20260621 07:08:58.260340 20772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:58.262889 20772 webserver.cc:460] Webserver started at http://127.1.188.194:36359/ using document root <none> and password file <none>
I20260621 07:08:58.263701 20772 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:58.263823 20772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:58.264182 20772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:58.267239 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "7ebcd3d22efa4d5c823d00a9265d788c"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "0b173f7946a9b4bd68ca715348f86f6f"
server_key_iv: "2836450e7c0b67eb86edd862dea06579"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.268177 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "7ebcd3d22efa4d5c823d00a9265d788c"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "0b173f7946a9b4bd68ca715348f86f6f"
server_key_iv: "2836450e7c0b67eb86edd862dea06579"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.274287 20772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:08:58.278178 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.279814 20772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 07:08:58.279999 20772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "7ebcd3d22efa4d5c823d00a9265d788c"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "0b173f7946a9b4bd68ca715348f86f6f"
server_key_iv: "2836450e7c0b67eb86edd862dea06579"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.280237 20772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:58.317466 20772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:58.318460 20772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:58.318814 20772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:58.319998 20772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:58.321647 20772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:58.321795 20772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.321939 20772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:58.322037 20772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.358958 20772 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:45713
I20260621 07:08:58.359056 20898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:45713 every 8 connection(s)
I20260621 07:08:58.360351 20772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:08:58.362247 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20772
I20260621 07:08:58.362421 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:08:58.363000 1779 external_mini_cluster.cc:1428] Setting key 213d15536c839e9742e05b7962d24545
I20260621 07:08:58.366451 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--builtin_ntp_servers=127.1.188.212:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:08:58.376935 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.363616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:40443 (local address 127.1.188.254:33613)
0621 07:08:58.363861 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.363871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.365211 (+ 1340us) server_negotiation.cc:407] Connection header received
0621 07:08:58.368037 (+ 2826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.368041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.368110 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.368208 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:58.369600 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.370568 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.373557 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.374005 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.374115 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.375967 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:58.375976 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:58.375983 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:58.376010 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:58.376056 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:58.376074 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:58.376181 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.376417 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.376532 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:08:58.377857 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.363956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33613 (local address 127.1.188.194:40443)
0621 07:08:58.365063 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:58.365107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:58.367732 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:58.368464 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:58.368477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:58.368597 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:58.369409 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.369435 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.370783 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.370793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.373367 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.373392 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.374266 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.374276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.374645 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.375525 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:58.375531 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:58.375589 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:58.375700 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:58.376193 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:58.376202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:58.376209 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 07:08:58.376358 (+ 149us) client_negotiation.cc:240] Negotiation successful
0621 07:08:58.376679 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":378,"threads_started":1}
I20260621 07:08:58.379513 20899 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33613
I20260621 07:08:58.379837 20899 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:58.380736 20899 heartbeater.cc:511] Master 127.1.188.254:33613 requested a full tablet report, sending...
I20260621 07:08:58.382283 20582 ts_manager.cc:194] Registered new tserver with Master: 7ebcd3d22efa4d5c823d00a9265d788c (127.1.188.194:45713)
I20260621 07:08:58.383096 20582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.194:40443
W20260621 07:08:58.697544 20902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:08:58.697945 20902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:08:58.698094 20902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:08:58.705855 20902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:08:58.705984 20902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:08:58.706156 20902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:08:58.715502 20902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:33613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:08:58.716833 20902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:08:58.718711 20902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:08:58.732606 20907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:58.732651 20908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:08:58.732566 20910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:08:58.733527 20902 server_base.cc:1034] running on GCE node
I20260621 07:08:58.734176 20902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:08:58.735174 20902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:08:58.736397 20902 hybrid_clock.cc:648] HybridClock initialized: now 1782025738736344 us; error 78 us; skew 500 ppm
I20260621 07:08:58.736750 20902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:08:58.739169 20902 webserver.cc:460] Webserver started at http://127.1.188.195:36661/ using document root <none> and password file <none>
I20260621 07:08:58.739996 20902 fs_manager.cc:359] Metadata directory not provided
I20260621 07:08:58.740118 20902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:08:58.740453 20902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:08:58.743472 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "0e3d1532fd9a4170ac274ea98454d093"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "3a05aa5f1924eccfca52c4c6cf16b292"
server_key_iv: "1ae36bb5d3470b8ce133969753a6c662"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.744287 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "0e3d1532fd9a4170ac274ea98454d093"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "3a05aa5f1924eccfca52c4c6cf16b292"
server_key_iv: "1ae36bb5d3470b8ce133969753a6c662"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.749402 20902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:08:58.752872 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.754244 20902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:08:58.754413 20902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "0e3d1532fd9a4170ac274ea98454d093"
format_stamp: "Formatted at 2026-06-21 07:08:58 on dist-test-slave-hlq1"
server_key: "3a05aa5f1924eccfca52c4c6cf16b292"
server_key_iv: "1ae36bb5d3470b8ce133969753a6c662"
server_key_version: "encryptionkey@0"
I20260621 07:08:58.754637 20902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:08:58.781980 20902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:08:58.782776 20902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:08:58.783030 20902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:08:58.784152 20902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:08:58.785890 20902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:08:58.785976 20902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.786062 20902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:08:58.786118 20902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:08:58.820799 20902 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:43723
I20260621 07:08:58.820891 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:43723 every 8 connection(s)
I20260621 07:08:58.822086 20902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:08:58.826411 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 20902
I20260621 07:08:58.826634 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:08:58.827250 1779 external_mini_cluster.cc:1428] Setting key 102f8075330ec6e5e078eeece53c98b8
I20260621 07:08:58.835249 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.824646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:33983 (local address 127.1.188.254:33613)
0621 07:08:58.824866 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.824871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.826257 (+ 1386us) server_negotiation.cc:407] Connection header received
0621 07:08:58.828622 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.828627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.828720 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.828834 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:58.830128 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.830879 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.832489 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.832877 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.833008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.834410 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:58.834415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:58.834418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:58.834436 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:58.834468 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:58.834479 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:58.834577 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.834798 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.834900 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260621 07:08:58.836326 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.825166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:33613 (local address 127.1.188.195:33983)
0621 07:08:58.826058 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:58.826103 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:58.828343 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:58.829066 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:58.829074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:58.829184 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:08:58.829968 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.829986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.831031 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.831036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.832343 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.832354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.833101 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.833108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.833342 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.834024 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:08:58.834027 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:08:58.834073 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:08:58.834149 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:08:58.834575 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:08:58.834587 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:08:58.834597 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 07:08:58.834797 (+ 200us) client_negotiation.cc:240] Negotiation successful
0621 07:08:58.835188 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":261,"threads_started":1}
I20260621 07:08:58.837756 21029 heartbeater.cc:347] Connected to a master server at 127.1.188.254:33613
I20260621 07:08:58.838102 21029 heartbeater.cc:464] Registering TS with master...
I20260621 07:08:58.838927 21029 heartbeater.cc:511] Master 127.1.188.254:33613 requested a full tablet report, sending...
I20260621 07:08:58.840523 20582 ts_manager.cc:194] Registered new tserver with Master: 0e3d1532fd9a4170ac274ea98454d093 (127.1.188.195:43723)
I20260621 07:08:58.841759 20582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.195:33983
I20260621 07:08:58.842636 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:08:58.853184 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.847172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.1.188.254:33613)
0621 07:08:58.847510 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.847520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.847895 (+ 375us) server_negotiation.cc:407] Connection header received
0621 07:08:58.848012 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.848016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.848075 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.848153 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:58.848620 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.849307 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.850780 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.851200 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.851293 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.852479 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:58.852483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:58.852487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:58.852503 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:58.852531 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:58.852542 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:58.852644 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.852815 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.852889 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260621 07:08:58.856972 20581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:08:58.864199 20581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:08:58.883991 20768 heartbeater.cc:503] Master 127.1.188.254:33613 was elected leader, sending a full tablet report...
I20260621 07:08:58.891676 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.876426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44338)
0621 07:08:58.878307 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:58.878360 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:58.878608 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:58.879696 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:58.879708 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:58.879756 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:58.880371 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.880389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.883552 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.883562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.887166 (+ 3604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.887191 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.890175 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.890186 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.890551 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.890640 (+ 89us) client_negotiation.cc:769] Sending connection context
0621 07:08:58.890796 (+ 156us) client_negotiation.cc:240] Negotiation successful
0621 07:08:58.891089 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"thread_start_us":1350,"threads_started":1}
I20260621 07:08:58.894096 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.876513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.1.188.193:39467)
0621 07:08:58.877556 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.877564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.878879 (+ 1315us) server_negotiation.cc:407] Connection header received
0621 07:08:58.879026 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.879039 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.879348 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.880688 (+ 1340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:58.880983 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.883284 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.887571 (+ 4287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.889865 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.892784 (+ 2919us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.893201 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.893300 (+ 99us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.893565 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":756,"threads_started":1}
I20260621 07:08:58.903618 20703 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7aac75b4d88803235b26d987f6c (DEFAULT_TABLE table=test-table [id=52b16d908d8b43e086ac897055aee3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:58.905905 20703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7aac75b4d88803235b26d987f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:58.910424 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.878724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:45713 (local address 127.0.0.1:32876)
0621 07:08:58.898380 (+ 19656us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:58.898407 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:58.898572 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:58.899968 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:58.899977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:58.900018 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:58.900570 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.900587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.903350 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.903360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.906690 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.906715 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.909487 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.909504 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.909665 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.909701 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:08:58.909843 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 07:08:58.909998 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19451,"thread_start_us":19259,"threads_started":1}
I20260621 07:08:58.910452 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.879647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.1.188.194:45713)
0621 07:08:58.897784 (+ 18137us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.897798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.898755 (+ 957us) server_negotiation.cc:407] Connection header received
0621 07:08:58.898900 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.898913 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.899280 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.899628 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:58.900832 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.903130 (+ 2298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.907314 (+ 4184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.908980 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.909115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.909387 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.909905 (+ 518us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.910084 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17878,"thread_start_us":17731,"threads_started":1}
I20260621 07:08:58.914781 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7aac75b4d88803235b26d987f6c (DEFAULT_TABLE table=test-table [id=52b16d908d8b43e086ac897055aee3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:58.916229 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.876987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:43723 (local address 127.0.0.1:44688)
0621 07:08:58.892365 (+ 15378us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:58.892394 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:58.892543 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:58.909014 (+ 16471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:58.909019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:58.909046 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:58.909276 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.909289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.912622 (+ 3333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.912628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.913873 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:58.913886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.915572 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.915579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:58.915799 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.915826 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 07:08:58.915884 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 07:08:58.915987 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15184,"thread_start_us":30080,"threads_started":1}
I20260621 07:08:58.916319 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7aac75b4d88803235b26d987f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:58.917260 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:58.877558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.1.188.195:43723)
0621 07:08:58.908135 (+ 30577us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:58.908153 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:58.908202 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:08:58.908314 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:58.908324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:58.908779 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:58.909012 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:58.910814 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.912455 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.914049 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:58.915311 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:58.916494 (+ 1183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:58.916787 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:58.916851 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 07:08:58.916973 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30398,"thread_start_us":30239,"threads_started":1}
I20260621 07:08:58.920873 20964 tablet_service.cc:1467] Processing CreateTablet for tablet 1002d7aac75b4d88803235b26d987f6c (DEFAULT_TABLE table=test-table [id=52b16d908d8b43e086ac897055aee3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:08:58.922304 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1002d7aac75b4d88803235b26d987f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:08:58.922545 21047 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Bootstrap starting.
I20260621 07:08:58.926525 21047 tablet_bootstrap.cc:654] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:58.927937 21047 log.cc:824] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:58.929605 21049 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Bootstrap starting.
I20260621 07:08:58.931216 21047 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: No bootstrap required, opened a new log
I20260621 07:08:58.931530 21047 ts_tablet_manager.cc:1397] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260621 07:08:58.933575 21049 tablet_bootstrap.cc:654] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:58.934949 21049 log.cc:824] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:58.938756 21049 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: No bootstrap required, opened a new log
I20260621 07:08:58.939152 21049 ts_tablet_manager.cc:1397] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260621 07:08:58.939704 21051 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Bootstrap starting.
I20260621 07:08:58.939765 21047 raft_consensus.cc:348] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.940311 21047 raft_consensus.cc:374] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:58.940449 21047 raft_consensus.cc:729] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e643052591a45f4802cdb35ca9fd599, State: Initialized, Role: FOLLOWER
I20260621 07:08:58.941066 21047 consensus_queue.cc:260] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.942921 21047 ts_tablet_manager.cc:1428] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260621 07:08:58.943650 21051 tablet_bootstrap.cc:654] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Neither blocks nor log segments found. Creating new log.
I20260621 07:08:58.944808 21051 log.cc:824] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Log is configured to *not* fsync() on all Append() calls
I20260621 07:08:58.948230 21051 tablet_bootstrap.cc:492] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: No bootstrap required, opened a new log
I20260621 07:08:58.948483 21051 ts_tablet_manager.cc:1397] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260621 07:08:58.950294 21049 raft_consensus.cc:348] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.950680 21049 raft_consensus.cc:374] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:58.950807 21049 raft_consensus.cc:729] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebcd3d22efa4d5c823d00a9265d788c, State: Initialized, Role: FOLLOWER
I20260621 07:08:58.951539 21049 consensus_queue.cc:260] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.952669 20899 heartbeater.cc:503] Master 127.1.188.254:33613 was elected leader, sending a full tablet report...
I20260621 07:08:58.953297 21049 ts_tablet_manager.cc:1428] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260621 07:08:58.956033 21051 raft_consensus.cc:348] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.956442 21051 raft_consensus.cc:374] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:08:58.956552 21051 raft_consensus.cc:729] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3d1532fd9a4170ac274ea98454d093, State: Initialized, Role: FOLLOWER
I20260621 07:08:58.957086 21051 consensus_queue.cc:260] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:58.958357 21029 heartbeater.cc:503] Master 127.1.188.254:33613 was elected leader, sending a full tablet report...
I20260621 07:08:58.958798 21051 ts_tablet_manager.cc:1428] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
W20260621 07:08:59.073886 21030 tablet.cc:2369] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:59.091831 21052 raft_consensus.cc:482] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:08:59.092182 21052 raft_consensus.cc:504] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:59.094056 21052 leader_election.cc:288] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ebcd3d22efa4d5c823d00a9265d788c (127.1.188.194:45713), 0e3d1532fd9a4170ac274ea98454d093 (127.1.188.195:43723)
I20260621 07:08:59.102191 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.094542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:49205 (local address 127.1.188.194:45713)
0621 07:08:59.094784 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.094792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.095784 (+ 992us) server_negotiation.cc:407] Connection header received
0621 07:08:59.095985 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.095990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.096056 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.096149 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:59.096771 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.097510 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.100357 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.101164 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.101302 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.101353 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.101803 (+ 450us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.101929 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 07:08:59.102192 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.094642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60355 (local address 127.1.188.195:43723)
0621 07:08:59.094866 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.094876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.095713 (+ 837us) server_negotiation.cc:407] Connection header received
0621 07:08:59.095946 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.095951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.096016 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.096125 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:08:59.096816 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.097635 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.100308 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.101296 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.101497 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.101572 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.101769 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.101927 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:08:59.102332 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.094593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:45713 (local address 127.1.188.193:49205)
0621 07:08:59.095672 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.095690 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:59.095836 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:59.096162 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:59.096166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:59.096189 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:59.096588 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:59.096597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.097676 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.097684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:59.100101 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:59.100114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.101371 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.101381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:59.101578 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.101623 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 07:08:59.101752 (+ 129us) client_negotiation.cc:240] Negotiation successful
0621 07:08:59.101917 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":34,"thread_start_us":641,"threads_started":1}
I20260621 07:08:59.102596 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.094543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:43723 (local address 127.1.188.193:60355)
0621 07:08:59.095570 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.095588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:08:59.095772 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:08:59.096140 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:08:59.096145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:08:59.096178 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:08:59.096664 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:59.096674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.097808 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.097814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:59.100098 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:08:59.100110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.101497 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.101502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:08:59.101611 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.101634 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 07:08:59.101705 (+ 71us) client_negotiation.cc:240] Negotiation successful
0621 07:08:59.101895 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":120,"thread_start_us":477,"threads_started":1}
I20260621 07:08:59.103456 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7aac75b4d88803235b26d987f6c" candidate_uuid: "5e643052591a45f4802cdb35ca9fd599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" is_pre_election: true
I20260621 07:08:59.103876 20984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7aac75b4d88803235b26d987f6c" candidate_uuid: "5e643052591a45f4802cdb35ca9fd599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3d1532fd9a4170ac274ea98454d093" is_pre_election: true
I20260621 07:08:59.104084 20854 raft_consensus.cc:2454] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e643052591a45f4802cdb35ca9fd599 in term 0.
I20260621 07:08:59.104423 20984 raft_consensus.cc:2454] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e643052591a45f4802cdb35ca9fd599 in term 0.
I20260621 07:08:59.104821 20657 leader_election.cc:302] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e643052591a45f4802cdb35ca9fd599, 7ebcd3d22efa4d5c823d00a9265d788c; no voters:
I20260621 07:08:59.105297 21052 raft_consensus.cc:2781] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:08:59.105424 21052 raft_consensus.cc:482] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:08:59.105538 21052 raft_consensus.cc:3037] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:59.107831 21052 raft_consensus.cc:504] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:59.108464 21052 leader_election.cc:288] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [CANDIDATE]: Term 1 election: Requested vote from peers 7ebcd3d22efa4d5c823d00a9265d788c (127.1.188.194:45713), 0e3d1532fd9a4170ac274ea98454d093 (127.1.188.195:43723)
I20260621 07:08:59.108978 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7aac75b4d88803235b26d987f6c" candidate_uuid: "5e643052591a45f4802cdb35ca9fd599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebcd3d22efa4d5c823d00a9265d788c"
I20260621 07:08:59.108978 20984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1002d7aac75b4d88803235b26d987f6c" candidate_uuid: "5e643052591a45f4802cdb35ca9fd599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3d1532fd9a4170ac274ea98454d093"
I20260621 07:08:59.109167 20854 raft_consensus.cc:3037] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:59.109203 20984 raft_consensus.cc:3037] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:08:59.111368 20854 raft_consensus.cc:2454] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e643052591a45f4802cdb35ca9fd599 in term 1.
I20260621 07:08:59.111694 20984 raft_consensus.cc:2454] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e643052591a45f4802cdb35ca9fd599 in term 1.
I20260621 07:08:59.111871 20657 leader_election.cc:302] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e643052591a45f4802cdb35ca9fd599, 7ebcd3d22efa4d5c823d00a9265d788c; no voters:
W20260621 07:08:59.111912 20769 tablet.cc:2369] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:59.112190 21052 raft_consensus.cc:2781] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 1 FOLLOWER]: Leader election won for term 1
W20260621 07:08:59.112349 20900 tablet.cc:2369] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:08:59.112967 21052 raft_consensus.cc:686] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 1 LEADER]: Becoming Leader. State: Replica: 5e643052591a45f4802cdb35ca9fd599, State: Running, Role: LEADER
I20260621 07:08:59.113440 21052 consensus_queue.cc:237] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } }
I20260621 07:08:59.117900 20581 catalog_manager.cc:5495] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e643052591a45f4802cdb35ca9fd599 (127.1.188.193). New cstate: current_term: 1 leader_uuid: "5e643052591a45f4802cdb35ca9fd599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e643052591a45f4802cdb35ca9fd599" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 } health_report { overall_health: UNKNOWN } } }
I20260621 07:08:59.158952 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.153360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.1.188.193:39467)
0621 07:08:59.153560 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.153567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.153618 (+ 51us) server_negotiation.cc:407] Connection header received
0621 07:08:59.153870 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.153874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.153968 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.154127 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:08:59.154555 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.155238 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.156760 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.157172 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.157306 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.157907 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:08:59.158383 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:08:59.158592 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.158678 (+ 86us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.158720 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:08:59.167649 20854 raft_consensus.cc:1261] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 1 FOLLOWER]: Refusing update from remote peer 5e643052591a45f4802cdb35ca9fd599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:59.167649 20984 raft_consensus.cc:1261] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 1 FOLLOWER]: Refusing update from remote peer 5e643052591a45f4802cdb35ca9fd599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:08:59.168684 21058 consensus_queue.cc:1035] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e3d1532fd9a4170ac274ea98454d093" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:59.169032 21052 consensus_queue.cc:1035] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebcd3d22efa4d5c823d00a9265d788c" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 45713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:08:59.180389 21061 mvcc.cc:205] Tried to move back new op lower bound from 7299177427619246080 to 7299177427412357120. Current Snapshot: MvccSnapshot[applied={T|T < 7299177427619246080}]
I20260621 07:08:59.181770 21062 mvcc.cc:205] Tried to move back new op lower bound from 7299177427619246080 to 7299177427412357120. Current Snapshot: MvccSnapshot[applied={T|T < 7299177427619246080}]
I20260621 07:08:59.186893 21063 mvcc.cc:205] Tried to move back new op lower bound from 7299177427619246080 to 7299177427412357120. Current Snapshot: MvccSnapshot[applied={T|T < 7299177427619246080}]
I20260621 07:08:59.195621 20582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:59.195888 20582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:08:59.199625 20582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 103c169eb7e247678392c2bc5b628c0f: Sending DeleteTablet for 3 replicas of tablet 1002d7aac75b4d88803235b26d987f6c
I20260621 07:08:59.200819 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 1002d7aac75b4d88803235b26d987f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:59 UTC) from {username='slave'} at 127.0.0.1:32876
I20260621 07:08:59.200794 20703 tablet_service.cc:1514] Processing DeleteTablet for tablet 1002d7aac75b4d88803235b26d987f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:59 UTC) from {username='slave'} at 127.0.0.1:44338
I20260621 07:08:59.200903 20964 tablet_service.cc:1514] Processing DeleteTablet for tablet 1002d7aac75b4d88803235b26d987f6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:08:59 UTC) from {username='slave'} at 127.0.0.1:44688
I20260621 07:08:59.202986 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20641
I20260621 07:08:59.204876 21073 tablet_replica.cc:330] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: stopping tablet replica
I20260621 07:08:59.205346 21073 raft_consensus.cc:2229] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:59.205917 21073 raft_consensus.cc:2258] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:59.217471 21073 ts_tablet_manager.cc:1905] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:59.220536 21072 tablet_replica.cc:330] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: stopping tablet replica
I20260621 07:08:59.220995 21072 raft_consensus.cc:2229] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:08:59.221714 21072 raft_consensus.cc:2258] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:59.224867 21073 ts_tablet_manager.cc:1918] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:59.225073 21073 log.cc:1196] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/1002d7aac75b4d88803235b26d987f6c
I20260621 07:08:59.225065 21072 ts_tablet_manager.cc:1905] T 1002d7aac75b4d88803235b26d987f6c P 5e643052591a45f4802cdb35ca9fd599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:59.225361 21071 tablet_replica.cc:330] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: stopping tablet replica
I20260621 07:08:59.225651 21073 ts_tablet_manager.cc:1939] T 1002d7aac75b4d88803235b26d987f6c P 0e3d1532fd9a4170ac274ea98454d093: Deleting consensus metadata
I20260621 07:08:59.225837 21071 raft_consensus.cc:2229] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:08:59.226403 21071 raft_consensus.cc:2258] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:08:59.227741 20566 catalog_manager.cc:4841] TS 0e3d1532fd9a4170ac274ea98454d093 (127.1.188.195:43723): tablet 1002d7aac75b4d88803235b26d987f6c (table test-table [id=52b16d908d8b43e086ac897055aee3c5]) successfully deleted
I20260621 07:08:59.228892 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.206421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.1.188.193:39467)
0621 07:08:59.207034 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.207042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.215450 (+ 8408us) server_negotiation.cc:407] Connection header received
0621 07:08:59.215972 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.215978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.216090 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.216193 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:59.216910 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.218054 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.222140 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.222802 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.222904 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.226903 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:59.226915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:59.226936 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:59.227812 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:59.227877 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:59.227903 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:59.227991 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.228373 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.228501 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489}
I20260621 07:08:59.229943 20763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:59.230890 21071 ts_tablet_manager.cc:1905] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:08:59.237252 21071 ts_tablet_manager.cc:1918] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:08:59.237425 21071 log.cc:1196] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/1002d7aac75b4d88803235b26d987f6c
I20260621 07:08:59.237936 21071 ts_tablet_manager.cc:1939] T 1002d7aac75b4d88803235b26d987f6c P 7ebcd3d22efa4d5c823d00a9265d788c: Deleting consensus metadata
I20260621 07:08:59.239620 20567 catalog_manager.cc:4841] TS 7ebcd3d22efa4d5c823d00a9265d788c (127.1.188.194:45713): tablet 1002d7aac75b4d88803235b26d987f6c (table test-table [id=52b16d908d8b43e086ac897055aee3c5]) successfully deleted
I20260621 07:08:59.327772 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20641
I20260621 07:08:59.344256 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20772
W20260621 07:08:59.344300 20567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.188.193:39467 (error 108)
I20260621 07:08:59.346534 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.345950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44366)
0621 07:08:59.346177 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.346375 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 07:08:59.346778 20567 catalog_manager.cc:4584] TS 5e643052591a45f4802cdb35ca9fd599 (127.1.188.193:39467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1002d7aac75b4d88803235b26d987f6c: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
I20260621 07:08:59.351593 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.345404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.1.188.194:45713)
0621 07:08:59.345580 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.345584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.345600 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:08:59.345655 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.345658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.345760 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.345870 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:59.346336 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.347235 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.348657 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.349039 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.349131 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.350182 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:59.350190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:59.350202 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:59.350784 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:59.350832 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:59.350851 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:59.350939 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.351076 (+ 137us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.351169 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:08:59.352316 20894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:59.400506 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.400035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44382)
0621 07:08:59.400241 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.400351 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 07:08:59.401614 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.401258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44388)
0621 07:08:59.401408 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.401481 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 07:08:59.444790 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20772
I20260621 07:08:59.461838 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20902
I20260621 07:08:59.462204 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.461685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44396)
0621 07:08:59.461935 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.462040 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:08:59.463356 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.462950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44404)
0621 07:08:59.463124 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.463191 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 07:08:59.468765 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.462655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.1.188.195:43723)
0621 07:08:59.462847 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 07:08:59.462853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:08:59.462873 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:08:59.463046 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:08:59.463052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:08:59.463159 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:08:59.463249 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:08:59.463737 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.464481 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.465760 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:08:59.466135 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:08:59.466234 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:08:59.467348 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:08:59.467355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:08:59.467369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:08:59.468021 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:08:59.468067 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:08:59.468087 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:08:59.468172 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 07:08:59.468337 (+ 165us) server_negotiation.cc:299] Negotiation successful
0621 07:08:59.468423 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 07:08:59.469373 21024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:59.544093 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.543607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44420)
0621 07:08:59.543846 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.543943 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260621 07:08:59.545193 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:08:59.544773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39467 (local address 127.0.0.1:44424)
0621 07:08:59.544979 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 07:08:59.545066 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260621 07:08:59.559284 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20902
I20260621 07:08:59.576138 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 20551
I20260621 07:08:59.577272 20612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:08:59.705873 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 20551
2026-06-21T07:08:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3034 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:08:59 dist-test-slave-hlq1 krb5kdc[21082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:08:59 dist-test-slave-hlq1 krb5kdc[21082](info): set up 2 sockets
Jun 21 07:08:59 dist-test-slave-hlq1 krb5kdc[21082](info): commencing operation
krb5kdc: starting...
W20260621 07:09:00.856184 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:01 dist-test-slave-hlq1 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:01.162993 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:38787
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:38787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:01.551807 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:01.552198 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:01.552358 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:01.560189 21095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:01.560359 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:01.560470 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:01.560583 21095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:01.560663 21095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:01.571211 21095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:38787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:38787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:01.572909 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:01.574512 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:01.587911 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:01.587891 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:01.588114 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:01.589365 21095 server_base.cc:1034] running on GCE node
I20260621 07:09:01.590270 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:01.592064 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:01.593314 21095 hybrid_clock.cc:648] HybridClock initialized: now 1782025741593257 us; error 49 us; skew 500 ppm
I20260621 07:09:01.593710 21095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:01 dist-test-slave-hlq1 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:01.597307 21095 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:01.598896 21095 webserver.cc:460] Webserver started at http://127.1.188.254:41525/ using document root <none> and password file <none>
I20260621 07:09:01.599810 21095 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:01.599958 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:01.600337 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:01.603842 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9bdd4338de174eb6901611bf3bfbcaa3"
format_stamp: "Formatted at 2026-06-21 07:09:01 on dist-test-slave-hlq1"
server_key: "1a9715a29e5bae7e0bfa36825f0af3aa"
server_key_iv: "fea43d25a4adecd8c0d54c16323382a2"
server_key_version: "encryptionkey@0"
I20260621 07:09:01.604669 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9bdd4338de174eb6901611bf3bfbcaa3"
format_stamp: "Formatted at 2026-06-21 07:09:01 on dist-test-slave-hlq1"
server_key: "1a9715a29e5bae7e0bfa36825f0af3aa"
server_key_iv: "fea43d25a4adecd8c0d54c16323382a2"
server_key_version: "encryptionkey@0"
I20260621 07:09:01.610379 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:09:01.614089 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:01.615610 21095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:09:01.615830 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9bdd4338de174eb6901611bf3bfbcaa3"
format_stamp: "Formatted at 2026-06-21 07:09:01 on dist-test-slave-hlq1"
server_key: "1a9715a29e5bae7e0bfa36825f0af3aa"
server_key_iv: "fea43d25a4adecd8c0d54c16323382a2"
server_key_version: "encryptionkey@0"
I20260621 07:09:01.616060 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:01.640339 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:01.645613 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:01.645942 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:01.666915 21095 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:38787
I20260621 07:09:01.666935 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:38787 every 8 connection(s)
I20260621 07:09:01.668632 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:01.673209 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:01.674499 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21095
I20260621 07:09:01.674721 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:01.675222 1779 external_mini_cluster.cc:1428] Setting key 30bd3f88b471845421d01ca87520d980
I20260621 07:09:01.690061 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: Bootstrap starting.
Jun 21 07:09:01 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:01.694233 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:01.695791 21162 log.cc:824] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:01.699754 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: No bootstrap required, opened a new log
I20260621 07:09:01.703078 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:01.677534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.1.188.254:38787)
0621 07:09:01.678442 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:01.678465 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:01.678528 (+ 63us) server_negotiation.cc:407] Connection header received
0621 07:09:01.680785 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:01.680816 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:01.681423 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:01.681936 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:01.683349 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:01.685865 (+ 2516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:01.687636 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:01.688851 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:01.689046 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:01.693846 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:01.693858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:01.693881 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:01.693936 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:01.698153 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:01.698913 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:01.698921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:01.698930 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:01.699146 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:01.699661 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:01.699666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:01.699669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:01.700456 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:01.700786 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:01.701130 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 07:09:01.701601 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":356,"threads_started":1}
I20260621 07:09:01.709386 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } }
I20260621 07:09:01.709779 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:01.709903 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bdd4338de174eb6901611bf3bfbcaa3, State: Initialized, Role: FOLLOWER
I20260621 07:09:01.710582 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } }
I20260621 07:09:01.710814 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:01.710912 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:01.711052 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:01.713251 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } }
I20260621 07:09:01.713719 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bdd4338de174eb6901611bf3bfbcaa3; no voters:
I20260621 07:09:01.714238 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:01.714412 21167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:01.715217 21167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [term 1 LEADER]: Becoming Leader. State: Replica: 9bdd4338de174eb6901611bf3bfbcaa3, State: Running, Role: LEADER
I20260621 07:09:01.715811 21167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } }
I20260621 07:09:01.716622 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:01.718770 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bdd4338de174eb6901611bf3bfbcaa3. Latest consensus state: current_term: 1 leader_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } } }
I20260621 07:09:01.719229 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:01.719189 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdd4338de174eb6901611bf3bfbcaa3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 38787 } } }
I20260621 07:09:01.719504 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:01.720520 21175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:01.725549 21175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:01.735245 21175 catalog_manager.cc:1347] Generated new cluster ID: 6391dd04daa44c98ad735e598ec51fd2
I20260621 07:09:01.735431 21175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:01.747969 21175 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:01.749354 21175 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:01.764317 21175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: Generated new TSK 0
I20260621 07:09:01.765493 21175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.188.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
Entry for principal kudu/127.1.188.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab.
I20260621 07:09:01.833097 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:02.185110 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:02.185412 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:02.185586 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:02.192895 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:02.193040 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:02.193208 21188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:09:02.203491 21188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.193.keytab
--principal=kudu/127.1.188.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:02.204790 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:02.206097 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:02.220337 21193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:02.220315 21194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:02.220285 21196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:02.224931 21188 server_base.cc:1034] running on GCE node
I20260621 07:09:02.225579 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:02.226751 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:02.227979 21188 hybrid_clock.cc:648] HybridClock initialized: now 1782025742227929 us; error 50 us; skew 500 ppm
I20260621 07:09:02.228339 21188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:02 dist-test-slave-hlq1 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:02.231627 21188 init.cc:377] Logged in from keytab as kudu/127.1.188.193@KRBTEST.COM (short username kudu)
I20260621 07:09:02.233091 21188 webserver.cc:460] Webserver started at http://127.1.188.193:36601/ using document root <none> and password file <none>
I20260621 07:09:02.233809 21188 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:02.233954 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:02.234272 21188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:02.237232 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "1e3cad1cf6334405b91d2ab30103c618"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "3cb621aa3e1b664b7ae93816de9de867"
server_key_iv: "fe70db8d3fd6e3cf5d7d90d8729138e7"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.238025 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "1e3cad1cf6334405b91d2ab30103c618"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "3cb621aa3e1b664b7ae93816de9de867"
server_key_iv: "fe70db8d3fd6e3cf5d7d90d8729138e7"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.243477 21188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 07:09:02.247071 21203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.248445 21188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:09:02.248622 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "1e3cad1cf6334405b91d2ab30103c618"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "3cb621aa3e1b664b7ae93816de9de867"
server_key_iv: "fe70db8d3fd6e3cf5d7d90d8729138e7"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.248831 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:02.307283 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:02.311661 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:02.311960 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:02.312924 21188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:02.314285 21188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:02.314388 21188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.314535 21188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:02.314608 21188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.347908 21188 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:42497
I20260621 07:09:02.347994 21315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:42497 every 8 connection(s)
I20260621 07:09:02.349211 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:09:02.351976 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21188
I20260621 07:09:02.352186 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:09:02.352985 1779 external_mini_cluster.cc:1428] Setting key 169c0b8014314c6150c3123cf4b7c24d
Jun 21 07:09:02 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.193@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:02.372362 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:02.351905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:36367 (local address 127.1.188.254:38787)
0621 07:09:02.354360 (+ 2455us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:02.354364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:02.354381 (+ 17us) server_negotiation.cc:407] Connection header received
0621 07:09:02.355953 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:02.355958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:02.356050 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:02.356172 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:02.357574 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.358722 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.360362 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.361108 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.361265 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:02.364725 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:02.364734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:02.364742 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:02.364786 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:02.368689 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:02.369560 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:02.369568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:02.369574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:02.369729 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:02.370182 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:02.370189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:02.370195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:02.370593 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:02.370752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:02.371593 (+ 841us) server_negotiation.cc:299] Negotiation successful
0621 07:09:02.371867 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2337,"thread_start_us":2238,"threads_started":1}
I20260621 07:09:02.373106 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:02.352480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38787 (local address 127.1.188.193:36367)
0621 07:09:02.353498 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:02.353553 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:02.355694 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:02.356349 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:02.356357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:02.356867 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:02.357403 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:02.357419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.358878 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.358882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:02.360219 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:02.360231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.361318 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.361322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:02.361513 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:02.362211 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:02.362232 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:02.364499 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:02.368884 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:02.368892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:02.368909 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:02.369414 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:02.369850 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:02.369853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:02.369855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:02.370050 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:02.370749 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:02.370756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:02.371098 (+ 342us) client_negotiation.cc:769] Sending connection context
0621 07:09:02.371390 (+ 292us) client_negotiation.cc:240] Negotiation successful
0621 07:09:02.371882 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":516,"threads_started":1}
I20260621 07:09:02.374843 21316 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38787
I20260621 07:09:02.375151 21316 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:02.376008 21316 heartbeater.cc:511] Master 127.1.188.254:38787 requested a full tablet report, sending...
I20260621 07:09:02.378445 21127 ts_manager.cc:194] Registered new tserver with Master: 1e3cad1cf6334405b91d2ab30103c618 (127.1.188.193:42497)
I20260621 07:09:02.380671 21127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.193@KRBTEST.COM'} at 127.1.188.193:36367
WARNING: no policy specified for kudu/127.1.188.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
Entry for principal kudu/127.1.188.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab.
I20260621 07:09:02.412595 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:02.735023 21322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:02.735327 21322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:02.735508 21322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:02.742671 21322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:02.742796 21322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:02.742909 21322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:09:02.752851 21322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.194.keytab
--principal=kudu/127.1.188.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:02.754215 21322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:02.755481 21322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:02.768355 21327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:02.768294 21330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:02.768307 21328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:02.769191 21322 server_base.cc:1034] running on GCE node
I20260621 07:09:02.769781 21322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:02.770714 21322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:02.771940 21322 hybrid_clock.cc:648] HybridClock initialized: now 1782025742771900 us; error 56 us; skew 500 ppm
I20260621 07:09:02.772250 21322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:02 dist-test-slave-hlq1 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:02.775372 21322 init.cc:377] Logged in from keytab as kudu/127.1.188.194@KRBTEST.COM (short username kudu)
I20260621 07:09:02.776912 21322 webserver.cc:460] Webserver started at http://127.1.188.194:43919/ using document root <none> and password file <none>
I20260621 07:09:02.777571 21322 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:02.777715 21322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:02.778030 21322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:02.780925 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "07cdd0463a044652bb29ec7c9a483d78"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "59721d7d50c6f480c68ef079a4e5b8e2"
server_key_iv: "d73ad400c19564f6433a7c4af0bd373c"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.781685 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "07cdd0463a044652bb29ec7c9a483d78"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "59721d7d50c6f480c68ef079a4e5b8e2"
server_key_iv: "d73ad400c19564f6433a7c4af0bd373c"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.786684 21322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 07:09:02.790088 21337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.791478 21322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:09:02.791651 21322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "07cdd0463a044652bb29ec7c9a483d78"
format_stamp: "Formatted at 2026-06-21 07:09:02 on dist-test-slave-hlq1"
server_key: "59721d7d50c6f480c68ef079a4e5b8e2"
server_key_iv: "d73ad400c19564f6433a7c4af0bd373c"
server_key_version: "encryptionkey@0"
I20260621 07:09:02.791865 21322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:02.814877 21322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:02.819113 21322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:02.819465 21322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:02.820462 21322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:02.821818 21322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:02.821890 21322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.822028 21322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:02.822086 21322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:02.853963 21322 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:35971
I20260621 07:09:02.854012 21449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:35971 every 8 connection(s)
I20260621 07:09:02.855280 21322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:09:02.856931 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21322
I20260621 07:09:02.857112 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:09:02.857666 1779 external_mini_cluster.cc:1428] Setting key 735837577aecdeaaeca4da538ecf92c8
Jun 21 07:09:02 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025742, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.194@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:02.876067 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:02.858000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:38943 (local address 127.1.188.254:38787)
0621 07:09:02.858179 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:02.858183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:02.859640 (+ 1457us) server_negotiation.cc:407] Connection header received
0621 07:09:02.861578 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:02.861586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:02.861677 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:02.861797 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:02.863258 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.864053 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.865613 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.866069 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.866202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:02.870248 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:02.870253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:02.870257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:02.870292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:02.873036 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:02.873775 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:02.873782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:02.873785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:02.873915 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:02.874338 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:02.874345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:02.874348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:02.874622 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:02.874780 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:02.875496 (+ 716us) server_negotiation.cc:299] Negotiation successful
0621 07:09:02.875723 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:09:02.877066 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:02.858556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38787 (local address 127.1.188.194:38943)
0621 07:09:02.859482 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:02.859536 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:02.861320 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:02.862021 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:02.862029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:02.862554 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:02.863087 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:02.863103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.864221 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.864225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:02.865461 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:02.865473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:02.866316 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:02.866324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:02.866552 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:02.867314 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:02.867349 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:02.870010 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:02.873191 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:02.873201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:02.873217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:02.873646 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:02.874027 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:02.874032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:02.874036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:02.874227 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:02.874763 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:02.874771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:02.875111 (+ 340us) client_negotiation.cc:769] Sending connection context
0621 07:09:02.875388 (+ 277us) client_negotiation.cc:240] Negotiation successful
0621 07:09:02.875954 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":343,"threads_started":1}
I20260621 07:09:02.878680 21450 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38787
I20260621 07:09:02.878990 21450 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:02.879812 21450 heartbeater.cc:511] Master 127.1.188.254:38787 requested a full tablet report, sending...
I20260621 07:09:02.881403 21127 ts_manager.cc:194] Registered new tserver with Master: 07cdd0463a044652bb29ec7c9a483d78 (127.1.188.194:35971)
WARNING: no policy specified for kudu/127.1.188.195@KRBTEST.COM; defaulting to no policy
I20260621 07:09:02.882652 21127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.194@KRBTEST.COM'} at 127.1.188.194:38943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
Entry for principal kudu/127.1.188.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab.
I20260621 07:09:02.914362 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:03.239807 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:03.240156 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:03.240338 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:03.247712 21455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:03.247867 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:03.248042 21455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:09:03.258230 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.195.keytab
--principal=kudu/127.1.188.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:03.259802 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:03.261166 21455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:03.274365 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:03.274492 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:03.274492 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:03.275377 21455 server_base.cc:1034] running on GCE node
I20260621 07:09:03.276012 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:03.276939 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:03.278160 21455 hybrid_clock.cc:648] HybridClock initialized: now 1782025743278111 us; error 61 us; skew 500 ppm
I20260621 07:09:03.278487 21455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:03 dist-test-slave-hlq1 krb5kdc[21082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025743, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:03.281826 21455 init.cc:377] Logged in from keytab as kudu/127.1.188.195@KRBTEST.COM (short username kudu)
I20260621 07:09:03.283272 21455 webserver.cc:460] Webserver started at http://127.1.188.195:42951/ using document root <none> and password file <none>
I20260621 07:09:03.284005 21455 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:03.284147 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:03.284469 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:03.287276 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "b90536a2752448df9556718f038823f0"
format_stamp: "Formatted at 2026-06-21 07:09:03 on dist-test-slave-hlq1"
server_key: "5a7393ce4c4d0900fee284d9ed9f7c81"
server_key_iv: "522b5230cd90224a9626448bcaab8641"
server_key_version: "encryptionkey@0"
I20260621 07:09:03.288121 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "b90536a2752448df9556718f038823f0"
format_stamp: "Formatted at 2026-06-21 07:09:03 on dist-test-slave-hlq1"
server_key: "5a7393ce4c4d0900fee284d9ed9f7c81"
server_key_iv: "522b5230cd90224a9626448bcaab8641"
server_key_version: "encryptionkey@0"
I20260621 07:09:03.293174 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:09:03.296613 21470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:03.297961 21455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:09:03.298135 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "b90536a2752448df9556718f038823f0"
format_stamp: "Formatted at 2026-06-21 07:09:03 on dist-test-slave-hlq1"
server_key: "5a7393ce4c4d0900fee284d9ed9f7c81"
server_key_iv: "522b5230cd90224a9626448bcaab8641"
server_key_version: "encryptionkey@0"
I20260621 07:09:03.298344 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:03.324860 21455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:03.329381 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:03.329694 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:03.330643 21455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:03.332026 21455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:03.332131 21455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:03.332284 21455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:03.332357 21455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:03.366499 21455 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:39777
I20260621 07:09:03.366588 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:39777 every 8 connection(s)
I20260621 07:09:03.367908 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:09:03.369228 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21455
I20260621 07:09:03.369412 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:09:03.369968 1779 external_mini_cluster.cc:1428] Setting key 7059b9e46667232ad4c8aef3c7b556ab
Jun 21 07:09:03 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025743, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.195@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:03.383723 21316 heartbeater.cc:503] Master 127.1.188.254:38787 was elected leader, sending a full tablet report...
I20260621 07:09:03.389228 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.370744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:58717 (local address 127.1.188.254:38787)
0621 07:09:03.370925 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.370929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.372464 (+ 1535us) server_negotiation.cc:407] Connection header received
0621 07:09:03.374316 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.374320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.374388 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.374475 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:03.376091 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.376897 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.378614 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.379017 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.379120 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.382852 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:03.382857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:03.382860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:03.382892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:03.386446 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.387165 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.387170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.387173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.387308 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.387750 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.387754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.387756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.388022 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:03.388131 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.388808 (+ 677us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.388970 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:09:03.390151 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.371470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:38787 (local address 127.1.188.195:58717)
0621 07:09:03.372309 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.372354 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.374057 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.374732 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.374740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.375259 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:03.375906 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.375923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.377059 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.377063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.378476 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.378489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.379262 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.379267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.379547 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.380277 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:03.380300 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:03.382620 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:03.386571 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:03.386577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:03.386590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:03.387050 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:03.387444 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:03.387449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:03.387452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:03.387644 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:03.388125 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:03.388130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:03.388448 (+ 318us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.388668 (+ 220us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.389050 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":345,"threads_started":1}
I20260621 07:09:03.391630 21583 heartbeater.cc:347] Connected to a master server at 127.1.188.254:38787
I20260621 07:09:03.391930 21583 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:03.392599 21583 heartbeater.cc:511] Master 127.1.188.254:38787 requested a full tablet report, sending...
I20260621 07:09:03.393994 21126 ts_manager.cc:194] Registered new tserver with Master: b90536a2752448df9556718f038823f0 (127.1.188.195:39777)
I20260621 07:09:03.394836 21126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.188.195@KRBTEST.COM'} at 127.1.188.195:58717
I20260621 07:09:03.398020 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:09:03.415012 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.402514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.1.188.254:38787)
0621 07:09:03.402712 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.402721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.403170 (+ 449us) server_negotiation.cc:407] Connection header received
0621 07:09:03.403318 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.403322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.403385 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.403534 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:03.404911 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.405656 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.406907 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.407348 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.407548 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.409965 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:03.409970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:03.409974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:03.410004 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:03.412579 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.413253 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.413257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.413259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.413387 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.413738 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.413742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.413744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.414064 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:03.414287 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.414607 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.414771 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 07:09:03.419601 21126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 07:09:03.426589 21126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:09:03.464982 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.438383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:39777 (local address 127.0.0.1:57712)
0621 07:09:03.451466 (+ 13083us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.451538 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.453141 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.453912 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.453932 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.453976 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:03.454510 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.454526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.457240 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.457248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.460447 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.460472 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.463573 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.463583 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.463945 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.464002 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.464139 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.464461 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12667,"mutex_wait_us":38,"spinlock_wait_cycles":9728,"thread_start_us":14247,"threads_started":1}
I20260621 07:09:03.465231 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.438638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.1.188.195:39777)
0621 07:09:03.451846 (+ 13208us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.451854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.451894 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:09:03.453453 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.453460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.453661 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.453853 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:03.454786 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.456977 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.460694 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.462867 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.463046 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.463392 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.464655 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.464828 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13053,"thread_start_us":12904,"threads_started":1}
I20260621 07:09:03.468339 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.438953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.1.188.194:35971)
0621 07:09:03.451300 (+ 12347us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.451315 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.452607 (+ 1292us) server_negotiation.cc:407] Connection header received
0621 07:09:03.453405 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.453413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.453668 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.453843 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:03.455510 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.459211 (+ 3701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.465923 (+ 6712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.467250 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.467423 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.467605 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.467897 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.468075 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12065,"thread_start_us":12841,"threads_started":1}
I20260621 07:09:03.469192 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.438429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:35971 (local address 127.0.0.1:49514)
0621 07:09:03.452489 (+ 14060us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.452506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.453141 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.454662 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.454670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.454723 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:03.455237 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.455252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.459460 (+ 4208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.459469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.464590 (+ 5121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.464605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.467501 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.467507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.467680 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.467712 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.467807 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.468712 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13922,"mutex_wait_us":673,"spinlock_wait_cycles":7808,"thread_start_us":13334,"threads_started":1}
I20260621 07:09:03.469455 21518 tablet_service.cc:1467] Processing CreateTablet for tablet bf1be878f53d4605b021345efeea3cf9 (DEFAULT_TABLE table=test-table [id=f206456deffe40d8ba5e34409359c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:03.471097 21518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1be878f53d4605b021345efeea3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:03.473188 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.452857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41782)
0621 07:09:03.464676 (+ 11819us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.464696 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.464822 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.465830 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.465838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.465866 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:03.466187 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.466197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.468052 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.468057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.470829 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.470842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.472340 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.472346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.472526 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.472562 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.472675 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.472804 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11675,"mutex_wait_us":537,"thread_start_us":11508,"threads_started":1}
I20260621 07:09:03.473223 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.453136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.1.188.193:42497)
0621 07:09:03.464403 (+ 11267us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.464413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.464966 (+ 553us) server_negotiation.cc:407] Connection header received
0621 07:09:03.465075 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.465083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.465345 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.465545 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:03.466383 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.467875 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.470998 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.472130 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.472281 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.472508 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.472741 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.472899 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11033,"thread_start_us":10819,"threads_started":1}
I20260621 07:09:03.473073 21385 tablet_service.cc:1467] Processing CreateTablet for tablet bf1be878f53d4605b021345efeea3cf9 (DEFAULT_TABLE table=test-table [id=f206456deffe40d8ba5e34409359c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:03.474522 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1be878f53d4605b021345efeea3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:03.476601 21251 tablet_service.cc:1467] Processing CreateTablet for tablet bf1be878f53d4605b021345efeea3cf9 (DEFAULT_TABLE table=test-table [id=f206456deffe40d8ba5e34409359c95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:03.477761 21251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1be878f53d4605b021345efeea3cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:03.488320 21601 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Bootstrap starting.
I20260621 07:09:03.490489 21602 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Bootstrap starting.
I20260621 07:09:03.492110 21603 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Bootstrap starting.
I20260621 07:09:03.493244 21601 tablet_bootstrap.cc:654] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:03.494587 21601 log.cc:824] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:03.494769 21602 tablet_bootstrap.cc:654] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:03.495582 21603 tablet_bootstrap.cc:654] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:03.495965 21602 log.cc:824] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:03.496834 21603 log.cc:824] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:03.498015 21601 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: No bootstrap required, opened a new log
I20260621 07:09:03.498373 21601 ts_tablet_manager.cc:1397] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260621 07:09:03.498837 21602 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: No bootstrap required, opened a new log
I20260621 07:09:03.499102 21602 ts_tablet_manager.cc:1397] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 07:09:03.499732 21603 tablet_bootstrap.cc:492] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: No bootstrap required, opened a new log
I20260621 07:09:03.499990 21603 ts_tablet_manager.cc:1397] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260621 07:09:03.507763 21602 raft_consensus.cc:348] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.508230 21602 raft_consensus.cc:374] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:03.508364 21602 raft_consensus.cc:729] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3cad1cf6334405b91d2ab30103c618, State: Initialized, Role: FOLLOWER
I20260621 07:09:03.509024 21602 consensus_queue.cc:260] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.509670 21601 raft_consensus.cc:348] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.510151 21601 raft_consensus.cc:374] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:03.510259 21601 raft_consensus.cc:729] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90536a2752448df9556718f038823f0, State: Initialized, Role: FOLLOWER
I20260621 07:09:03.510861 21602 ts_tablet_manager.cc:1428] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Time spent starting tablet: real 0.012s user 0.006s sys 0.008s
I20260621 07:09:03.510847 21601 consensus_queue.cc:260] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.511963 21583 heartbeater.cc:503] Master 127.1.188.254:38787 was elected leader, sending a full tablet report...
I20260621 07:09:03.512583 21601 ts_tablet_manager.cc:1428] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260621 07:09:03.512912 21603 raft_consensus.cc:348] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.513384 21603 raft_consensus.cc:374] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:03.513510 21603 raft_consensus.cc:729] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdd0463a044652bb29ec7c9a483d78, State: Initialized, Role: FOLLOWER
I20260621 07:09:03.514081 21603 consensus_queue.cc:260] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.515205 21450 heartbeater.cc:503] Master 127.1.188.254:38787 was elected leader, sending a full tablet report...
I20260621 07:09:03.515843 21603 ts_tablet_manager.cc:1428] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20260621 07:09:03.537385 21607 raft_consensus.cc:482] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:09:03.537719 21607 raft_consensus.cc:504] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.539537 21607 leader_election.cc:288] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b90536a2752448df9556718f038823f0 (127.1.188.195:39777), 07cdd0463a044652bb29ec7c9a483d78 (127.1.188.194:35971)
I20260621 07:09:03.546701 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.539938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45915 (local address 127.1.188.194:35971)
0621 07:09:03.540172 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.540187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.540894 (+ 707us) server_negotiation.cc:407] Connection header received
0621 07:09:03.541129 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.541135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.541225 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.541523 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:03.542072 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.542828 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.544889 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.545691 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.545897 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.545979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.546269 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.546407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:09:03.546743 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.540039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:45763 (local address 127.1.188.195:39777)
0621 07:09:03.540256 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.540260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.541358 (+ 1098us) server_negotiation.cc:407] Connection header received
0621 07:09:03.541641 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.541645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.541707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.541813 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:03.542389 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.543145 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.545018 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.545780 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.545990 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.546067 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.546348 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.546485 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:09:03.546838 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.540440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:39777 (local address 127.1.188.193:45763)
0621 07:09:03.541161 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.541175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.541291 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.541835 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.541840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.541871 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:03.542221 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.542228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.543293 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.543301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.544847 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.544859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.546021 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.546027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.546181 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.546205 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.546316 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.546457 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":421,"threads_started":1}
I20260621 07:09:03.546881 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.539996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:35971 (local address 127.1.188.193:45915)
0621 07:09:03.540774 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.540791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:03.540984 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:03.541538 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:03.541542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:03.541569 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:03.541926 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.541938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.542972 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.542977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.544711 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:03.544724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.545921 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.545928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:03.546096 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.546124 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 07:09:03.546241 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 07:09:03.546396 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":2,"thread_start_us":430,"threads_started":1}
I20260621 07:09:03.547964 21538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1be878f53d4605b021345efeea3cf9" candidate_uuid: "1e3cad1cf6334405b91d2ab30103c618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90536a2752448df9556718f038823f0" is_pre_election: true
I20260621 07:09:03.547957 21405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1be878f53d4605b021345efeea3cf9" candidate_uuid: "1e3cad1cf6334405b91d2ab30103c618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdd0463a044652bb29ec7c9a483d78" is_pre_election: true
I20260621 07:09:03.548470 21538 raft_consensus.cc:2454] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3cad1cf6334405b91d2ab30103c618 in term 0.
I20260621 07:09:03.548470 21405 raft_consensus.cc:2454] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e3cad1cf6334405b91d2ab30103c618 in term 0.
I20260621 07:09:03.549188 21207 leader_election.cc:302] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdd0463a044652bb29ec7c9a483d78, 1e3cad1cf6334405b91d2ab30103c618, b90536a2752448df9556718f038823f0; no voters:
I20260621 07:09:03.549661 21607 raft_consensus.cc:2781] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:09:03.549854 21607 raft_consensus.cc:482] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:09:03.549970 21607 raft_consensus.cc:3037] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:03.552490 21607 raft_consensus.cc:504] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.553316 21607 leader_election.cc:288] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [CANDIDATE]: Term 1 election: Requested vote from peers b90536a2752448df9556718f038823f0 (127.1.188.195:39777), 07cdd0463a044652bb29ec7c9a483d78 (127.1.188.194:35971)
I20260621 07:09:03.553683 21538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1be878f53d4605b021345efeea3cf9" candidate_uuid: "1e3cad1cf6334405b91d2ab30103c618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90536a2752448df9556718f038823f0"
I20260621 07:09:03.553807 21405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1be878f53d4605b021345efeea3cf9" candidate_uuid: "1e3cad1cf6334405b91d2ab30103c618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdd0463a044652bb29ec7c9a483d78"
I20260621 07:09:03.553905 21538 raft_consensus.cc:3037] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:03.554046 21405 raft_consensus.cc:3037] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:03.556273 21538 raft_consensus.cc:2454] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3cad1cf6334405b91d2ab30103c618 in term 1.
I20260621 07:09:03.556318 21405 raft_consensus.cc:2454] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e3cad1cf6334405b91d2ab30103c618 in term 1.
I20260621 07:09:03.556692 21205 leader_election.cc:302] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e3cad1cf6334405b91d2ab30103c618, b90536a2752448df9556718f038823f0; no voters:
I20260621 07:09:03.557003 21607 raft_consensus.cc:2781] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:03.557482 21607 raft_consensus.cc:686] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 1 LEADER]: Becoming Leader. State: Replica: 1e3cad1cf6334405b91d2ab30103c618, State: Running, Role: LEADER
I20260621 07:09:03.557960 21607 consensus_queue.cc:237] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } }
I20260621 07:09:03.562774 21124 catalog_manager.cc:5495] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e3cad1cf6334405b91d2ab30103c618 (127.1.188.193). New cstate: current_term: 1 leader_uuid: "1e3cad1cf6334405b91d2ab30103c618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e3cad1cf6334405b91d2ab30103c618" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 42497 } health_report { overall_health: HEALTHY } } }
I20260621 07:09:03.589040 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.583646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.1.188.193:42497)
0621 07:09:03.583855 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.583861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.583935 (+ 74us) server_negotiation.cc:407] Connection header received
0621 07:09:03.584189 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.584193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.584281 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.584455 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:03.584809 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.585539 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.586899 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.587284 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.587440 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.587951 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:03.588453 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:03.588628 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.588688 (+ 60us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.588769 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:09:03.597060 21405 raft_consensus.cc:1261] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 1 FOLLOWER]: Refusing update from remote peer 1e3cad1cf6334405b91d2ab30103c618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:03.597061 21538 raft_consensus.cc:1261] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 1 FOLLOWER]: Refusing update from remote peer 1e3cad1cf6334405b91d2ab30103c618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:03.597987 21612 consensus_queue.cc:1035] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdd0463a044652bb29ec7c9a483d78" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 35971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:03.598418 21607 consensus_queue.cc:1035] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90536a2752448df9556718f038823f0" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 39777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 07:09:03.601773 21317 tablet.cc:2369] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 07:09:03.607241 21451 tablet.cc:2369] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:03.634404 21125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:09:03.634718 21125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:09:03.638394 21125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bdd4338de174eb6901611bf3bfbcaa3: Sending DeleteTablet for 3 replicas of tablet bf1be878f53d4605b021345efeea3cf9
I20260621 07:09:03.639566 21518 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1be878f53d4605b021345efeea3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:03 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:57712
I20260621 07:09:03.639923 21251 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1be878f53d4605b021345efeea3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:03 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:41782
I20260621 07:09:03.640666 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21188
I20260621 07:09:03.640797 21385 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1be878f53d4605b021345efeea3cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:03 UTC) from {username='kudu', principal='kudu/127.1.188.254@KRBTEST.COM'} at 127.0.0.1:49514
I20260621 07:09:03.652022 21627 tablet_replica.cc:330] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: stopping tablet replica
I20260621 07:09:03.652639 21627 raft_consensus.cc:2229] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:09:03.653872 21626 tablet_replica.cc:330] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: stopping tablet replica
I20260621 07:09:03.653445 21627 raft_consensus.cc:2258] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:03.654466 21626 raft_consensus.cc:2229] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:09:03.655512 21626 raft_consensus.cc:2258] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:03.657013 21625 tablet_replica.cc:330] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: stopping tablet replica
I20260621 07:09:03.657409 21625 raft_consensus.cc:2229] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:09:03.657863 21625 raft_consensus.cc:2258] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:03.659600 21626 ts_tablet_manager.cc:1905] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:09:03.659773 21625 ts_tablet_manager.cc:1905] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:09:03.660809 21627 ts_tablet_manager.cc:1905] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 07:09:03 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.193@KRBTEST.COM
I20260621 07:09:03.667624 21627 ts_tablet_manager.cc:1918] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:03.667783 21627 log.cc:1196] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/bf1be878f53d4605b021345efeea3cf9
I20260621 07:09:03.668222 21625 ts_tablet_manager.cc:1918] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:03.668430 21625 log.cc:1196] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/bf1be878f53d4605b021345efeea3cf9
I20260621 07:09:03.669070 21625 ts_tablet_manager.cc:1939] T bf1be878f53d4605b021345efeea3cf9 P b90536a2752448df9556718f038823f0: Deleting consensus metadata
I20260621 07:09:03.671051 21112 catalog_manager.cc:4841] TS b90536a2752448df9556718f038823f0 (127.1.188.195:39777): tablet bf1be878f53d4605b021345efeea3cf9 (table test-table [id=f206456deffe40d8ba5e34409359c95d]) successfully deleted
I20260621 07:09:03.673431 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.641861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.1.188.193:42497)
0621 07:09:03.642095 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.642104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.651135 (+ 9031us) server_negotiation.cc:407] Connection header received
0621 07:09:03.651477 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.651490 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.651652 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.651803 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:03.656573 (+ 4770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.657488 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.659247 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.659977 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.660088 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.665543 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:03.665558 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:03.665574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:03.665617 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:03.670224 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.671116 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.671125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.671129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.671328 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.671759 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.671764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.671767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.672126 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:03.672452 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.672783 (+ 331us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.673050 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 07:09:03.674224 21311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:03.676803 21627 ts_tablet_manager.cc:1939] T bf1be878f53d4605b021345efeea3cf9 P 07cdd0463a044652bb29ec7c9a483d78: Deleting consensus metadata
I20260621 07:09:03.678737 21114 catalog_manager.cc:4841] TS 07cdd0463a044652bb29ec7c9a483d78 (127.1.188.194:35971): tablet bf1be878f53d4605b021345efeea3cf9 (table test-table [id=f206456deffe40d8ba5e34409359c95d]) successfully deleted
I20260621 07:09:03.772249 21626 ts_tablet_manager.cc:1918] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:03.772624 21626 log.cc:1196] T bf1be878f53d4605b021345efeea3cf9 P 1e3cad1cf6334405b91d2ab30103c618: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/bf1be878f53d4605b021345efeea3cf9
I20260621 07:09:03.772878 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21188
I20260621 07:09:03.792951 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21322
W20260621 07:09:03.792896 21114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.188.193:42497 (error 108)
I20260621 07:09:03.794857 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.794369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41806)
0621 07:09:03.794549 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.794667 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 07:09:03.795107 21114 catalog_manager.cc:4584] TS 1e3cad1cf6334405b91d2ab30103c618 (127.1.188.193:42497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf1be878f53d4605b021345efeea3cf9: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Jun 21 07:09:03 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.194@KRBTEST.COM
I20260621 07:09:03.809114 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.793830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49524 (local address 127.1.188.194:35971)
0621 07:09:03.794005 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.794009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.794120 (+ 111us) server_negotiation.cc:407] Connection header received
0621 07:09:03.794357 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.794363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.794462 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.794572 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:03.796236 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.797034 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.798304 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.798701 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.798800 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.803115 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:03.803128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:03.803146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:03.803183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:03.806468 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.807112 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.807119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.807122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.807303 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.807686 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.807691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.807695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.808081 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:03.808330 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.808552 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.808729 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 07:09:03.809861 21445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:03.849072 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.848560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41818)
0621 07:09:03.848808 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.848912 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260621 07:09:03.850062 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.849763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41826)
0621 07:09:03.849884 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.849961 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 07:09:03.900725 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21322
I20260621 07:09:03.910602 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.910098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41836)
0621 07:09:03.910334 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.910436 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 07:09:03.911813 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.911497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41838)
0621 07:09:03.911633 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.911707 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260621 07:09:03.918586 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21455
Jun 21 07:09:03 dist-test-slave-hlq1 krb5kdc[21082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.195@KRBTEST.COM
I20260621 07:09:03.934051 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.919363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.1.188.195:39777)
0621 07:09:03.919582 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:03.919588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:03.919611 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:09:03.919698 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:03.919703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:03.919775 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:03.919896 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:03.921236 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.921950 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.923249 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:03.923709 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:03.923839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:03.928285 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:03.928299 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:03.928316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:03.928350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:03.931143 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.931954 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.931962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.931964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.932105 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:03.932503 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:03.932507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:03.932512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:03.932866 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:03.933177 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:03.933497 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 07:09:03.933732 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260621 07:09:03.934769 21578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:03.992749 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.992281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41854)
0621 07:09:03.992500 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.992593 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 07:09:03.993793 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:03.993438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:42497 (local address 127.0.0.1:41856)
0621 07:09:03.993624 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:03.993690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:42497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260621 07:09:04.025653 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21455
I20260621 07:09:04.042831 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21095
I20260621 07:09:04.044000 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:04.160028 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21095
2026-06-21T07:09:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4450 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T07:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:04Z Disabled control of system clock
I20260621 07:09:04.236668 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41233
--webserver_interface=127.1.188.254
--webserver_port=0
--builtin_ntp_servers=127.1.188.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:09:04.558789 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:04.559055 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:04.559204 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:04.566747 21637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:04.566836 21637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:04.566923 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:04.567018 21637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:04.567077 21637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:04.576440 21637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:04.577718 21637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:04.579439 21637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:04.590777 21642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:04.590858 21645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:04.590885 21643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:04.592010 21637 server_base.cc:1034] running on GCE node
I20260621 07:09:04.592757 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:04.594120 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:04.595386 21637 hybrid_clock.cc:648] HybridClock initialized: now 1782025744595348 us; error 49 us; skew 500 ppm
I20260621 07:09:04.595788 21637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:09:04.598240 21637 webserver.cc:460] Webserver started at http://127.1.188.254:39457/ using document root <none> and password file <none>
I20260621 07:09:04.598932 21637 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:04.599074 21637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:04.599466 21637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:04.602330 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b"
format_stamp: "Formatted at 2026-06-21 07:09:04 on dist-test-slave-hlq1"
server_key: "9a8a132a50c6a7530cdca776ef697ea3"
server_key_iv: "e42db5542ae8f7dfac5ae0079661af37"
server_key_version: "encryptionkey@0"
I20260621 07:09:04.603179 21637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b"
format_stamp: "Formatted at 2026-06-21 07:09:04 on dist-test-slave-hlq1"
server_key: "9a8a132a50c6a7530cdca776ef697ea3"
server_key_iv: "e42db5542ae8f7dfac5ae0079661af37"
server_key_version: "encryptionkey@0"
I20260621 07:09:04.608278 21637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:09:04.611570 21651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:04.612980 21637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:09:04.613166 21637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b"
format_stamp: "Formatted at 2026-06-21 07:09:04 on dist-test-slave-hlq1"
server_key: "9a8a132a50c6a7530cdca776ef697ea3"
server_key_iv: "e42db5542ae8f7dfac5ae0079661af37"
server_key_version: "encryptionkey@0"
I20260621 07:09:04.613379 21637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:04.638252 21637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:04.639066 21637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:04.639364 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:04.658735 21637 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41233
I20260621 07:09:04.658768 21702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41233 every 8 connection(s)
I20260621 07:09:04.660076 21637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:04.663782 21703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:04.668434 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21637
I20260621 07:09:04.668638 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:04.669155 1779 external_mini_cluster.cc:1428] Setting key b0a039007aec8d7926f68d5cc5435489
I20260621 07:09:04.675941 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: Bootstrap starting.
I20260621 07:09:04.679703 21703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:04.681005 21703 log.cc:824] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:04.684620 21703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: No bootstrap required, opened a new log
I20260621 07:09:04.685213 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:04.671113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.1.188.254:41233)
0621 07:09:04.672276 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:04.672290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:04.672345 (+ 55us) server_negotiation.cc:407] Connection header received
0621 07:09:04.674769 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:04.674798 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:04.675369 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:04.675849 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:04.676393 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:04.678393 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:04.679897 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:04.680941 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:04.681134 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:04.682364 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:04.682371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:04.682385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:04.683114 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:04.683212 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:04.683239 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:04.683362 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:04.683654 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:09:04.683987 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":395,"threads_started":1}
I20260621 07:09:04.692793 21703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } }
I20260621 07:09:04.693245 21703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:04.693408 21703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae8d116ac1444c5afdc5ac5ec91e66b, State: Initialized, Role: FOLLOWER
I20260621 07:09:04.694010 21703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } }
I20260621 07:09:04.694224 21703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:04.694331 21703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:04.694450 21703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:04.696625 21703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } }
I20260621 07:09:04.697114 21703 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae8d116ac1444c5afdc5ac5ec91e66b; no voters:
I20260621 07:09:04.697681 21703 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:04.697911 21708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:04.698633 21708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [term 1 LEADER]: Becoming Leader. State: Replica: 4ae8d116ac1444c5afdc5ac5ec91e66b, State: Running, Role: LEADER
I20260621 07:09:04.699208 21708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } }
I20260621 07:09:04.700121 21703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:04.702152 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae8d116ac1444c5afdc5ac5ec91e66b. Latest consensus state: current_term: 1 leader_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } } }
I20260621 07:09:04.702527 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:04.702286 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae8d116ac1444c5afdc5ac5ec91e66b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41233 } } }
I20260621 07:09:04.702862 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:04.708319 21713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:04.714002 21713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:04.723141 21713 catalog_manager.cc:1347] Generated new cluster ID: 49b67233dcd54b7fb4288634414b3719
I20260621 07:09:04.723471 21713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:04.737396 21713 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:04.738781 21713 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:04.744799 21713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: Generated new TSK 0
I20260621 07:09:04.745638 21713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:04.758008 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.193:0
--local_ip_for_outbound_sockets=127.1.188.193
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--builtin_ntp_servers=127.1.188.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 07:09:05.093887 21727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:05.094149 21727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:05.094271 21727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:05.101513 21727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:05.101612 21727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:05.101718 21727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.193
I20260621 07:09:05.110827 21727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.188.193
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.193
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:05.112187 21727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:05.113977 21727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:05.126992 21732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:05.127070 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:05.127136 21733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:05.128296 21727 server_base.cc:1034] running on GCE node
I20260621 07:09:05.128912 21727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:05.129936 21727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:05.131182 21727 hybrid_clock.cc:648] HybridClock initialized: now 1782025745131121 us; error 69 us; skew 500 ppm
I20260621 07:09:05.131558 21727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:09:05.138305 21727 webserver.cc:460] Webserver started at http://127.1.188.193:40457/ using document root <none> and password file <none>
I20260621 07:09:05.139068 21727 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:05.139206 21727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:05.139612 21727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:05.142433 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data/instance:
uuid: "3d30af9f5f8247a788087122a969e549"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "93604d49cdb0b3a00ec2a3d54d25e55d"
server_key_iv: "8204d2458797ae3e46465b59cc7eafca"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.143293 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance:
uuid: "3d30af9f5f8247a788087122a969e549"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "93604d49cdb0b3a00ec2a3d54d25e55d"
server_key_iv: "8204d2458797ae3e46465b59cc7eafca"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.148315 21727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:09:05.151868 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.153249 21727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:09:05.153434 21727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal
uuid: "3d30af9f5f8247a788087122a969e549"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "93604d49cdb0b3a00ec2a3d54d25e55d"
server_key_iv: "8204d2458797ae3e46465b59cc7eafca"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.153678 21727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:05.166805 21727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:05.167647 21727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:05.167949 21727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:05.168896 21727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:05.170344 21727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:05.170432 21727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.170588 21727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:05.170696 21727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.204123 21727 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.193:39431
I20260621 07:09:05.204169 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.193:39431 every 8 connection(s)
I20260621 07:09:05.205458 21727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/data/info.pb
I20260621 07:09:05.207151 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21727
I20260621 07:09:05.207325 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal/instance
I20260621 07:09:05.207930 1779 external_mini_cluster.cc:1428] Setting key b94a6763e79a998a24e889ff670fcf77
I20260621 07:09:05.211061 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.194:0
--local_ip_for_outbound_sockets=127.1.188.194
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--builtin_ntp_servers=127.1.188.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:09:05.220402 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:05.208876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.193:60697 (local address 127.1.188.254:41233)
0621 07:09:05.209343 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:05.209347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:05.210986 (+ 1639us) server_negotiation.cc:407] Connection header received
0621 07:09:05.212862 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:05.212871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:05.212960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:05.213080 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:05.214197 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.215344 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.217121 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.217928 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.218071 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:05.219547 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:05.219552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:05.219556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:05.219575 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:05.219620 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:05.219632 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:05.219730 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:05.220040 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 07:09:05.220127 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":269,"threads_started":1}
I20260621 07:09:05.221904 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:05.209591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41233 (local address 127.1.188.193:60697)
0621 07:09:05.210799 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:05.210848 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:05.212601 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:05.213263 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:05.213271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:05.213354 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:05.214030 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:05.214049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.215529 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.215533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:05.216933 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:05.216947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.218180 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.218188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:05.218402 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:05.219171 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:09:05.219175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:05.219233 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:09:05.219340 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:05.219758 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:05.219769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:05.219778 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 07:09:05.219994 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 07:09:05.220427 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":419,"threads_started":1}
I20260621 07:09:05.223532 21854 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41233
I20260621 07:09:05.223875 21854 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:05.224711 21854 heartbeater.cc:511] Master 127.1.188.254:41233 requested a full tablet report, sending...
I20260621 07:09:05.226753 21668 ts_manager.cc:194] Registered new tserver with Master: 3d30af9f5f8247a788087122a969e549 (127.1.188.193:39431)
I20260621 07:09:05.228327 21668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.193:60697
W20260621 07:09:05.534150 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:05.534483 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:05.534644 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:05.541935 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:05.542068 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:05.542258 21858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.194
I20260621 07:09:05.553134 21858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.188.194
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.194
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:05.554948 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:05.557366 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:05.572365 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:05.572306 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:05.572279 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:05.573282 21858 server_base.cc:1034] running on GCE node
I20260621 07:09:05.573863 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:05.574770 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:05.575990 21858 hybrid_clock.cc:648] HybridClock initialized: now 1782025745575928 us; error 47 us; skew 500 ppm
I20260621 07:09:05.576300 21858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:09:05.578528 21858 webserver.cc:460] Webserver started at http://127.1.188.194:46379/ using document root <none> and password file <none>
I20260621 07:09:05.579197 21858 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:05.579319 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:05.579663 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:05.582384 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data/instance:
uuid: "165077c0cfb34d4589bb4db169ae964e"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "8c52f250b5f208d971031a60ff4da17e"
server_key_iv: "f8e9135e919f09fc9904c596b96a3404"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.583155 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance:
uuid: "165077c0cfb34d4589bb4db169ae964e"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "8c52f250b5f208d971031a60ff4da17e"
server_key_iv: "f8e9135e919f09fc9904c596b96a3404"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.588675 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 07:09:05.592811 21872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.594103 21858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260621 07:09:05.594327 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal
uuid: "165077c0cfb34d4589bb4db169ae964e"
format_stamp: "Formatted at 2026-06-21 07:09:05 on dist-test-slave-hlq1"
server_key: "8c52f250b5f208d971031a60ff4da17e"
server_key_iv: "f8e9135e919f09fc9904c596b96a3404"
server_key_version: "encryptionkey@0"
I20260621 07:09:05.594558 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:05.644651 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:05.645506 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:05.645819 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:05.646821 21858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:05.648442 21858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:05.648576 21858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.648705 21858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:05.648792 21858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:05.681732 21858 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.194:40555
I20260621 07:09:05.681777 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.194:40555 every 8 connection(s)
I20260621 07:09:05.683187 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/data/info.pb
I20260621 07:09:05.689368 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21858
I20260621 07:09:05.689556 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal/instance
I20260621 07:09:05.690008 1779 external_mini_cluster.cc:1428] Setting key a678d87a9fd822f35b29304ad5678b54
I20260621 07:09:05.692747 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.188.195:0
--local_ip_for_outbound_sockets=127.1.188.195
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--builtin_ntp_servers=127.1.188.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 07:09:05.696182 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:05.685854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:37345 (local address 127.1.188.254:41233)
0621 07:09:05.686036 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:05.686040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:05.687477 (+ 1437us) server_negotiation.cc:407] Connection header received
0621 07:09:05.689353 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:05.689357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:05.689422 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:05.689543 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:05.690605 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.691565 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.693146 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.693902 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.694049 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:05.695281 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:05.695289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:05.695297 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:05.695321 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:05.695364 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:05.695381 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:05.695493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:05.695676 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 07:09:05.695790 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:09:05.696918 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:05.686389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41233 (local address 127.1.188.194:37345)
0621 07:09:05.687291 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:05.687338 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:05.689003 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:05.689776 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:05.689783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:05.689861 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:05.690447 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:05.690464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.691752 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.691755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:05.692985 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:05.692999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:05.694093 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:05.694097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:05.694293 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:05.694990 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:09:05.694993 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:05.695035 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:09:05.695103 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:05.695489 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:05.695496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:05.695502 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 07:09:05.695634 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 07:09:05.695924 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":256,"threads_started":1}
I20260621 07:09:05.698541 21985 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41233
I20260621 07:09:05.698853 21985 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:05.699754 21985 heartbeater.cc:511] Master 127.1.188.254:41233 requested a full tablet report, sending...
I20260621 07:09:05.701366 21668 ts_manager.cc:194] Registered new tserver with Master: 165077c0cfb34d4589bb4db169ae964e (127.1.188.194:40555)
I20260621 07:09:05.702275 21668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.194:37345
W20260621 07:09:06.054297 21988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:06.054680 21988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:06.054855 21988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:06.062665 21988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:06.062814 21988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:06.063009 21988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.188.195
I20260621 07:09:06.072662 21988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.188.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.188.195
--webserver_port=0
--tserver_master_addrs=127.1.188.254:41233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.188.195
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:06.074272 21988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:06.076452 21988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:06.092729 21993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:06.092630 21994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:06.092667 21996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:06.093798 21988 server_base.cc:1034] running on GCE node
I20260621 07:09:06.094499 21988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:06.095695 21988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:06.096942 21988 hybrid_clock.cc:648] HybridClock initialized: now 1782025746096877 us; error 67 us; skew 500 ppm
I20260621 07:09:06.097334 21988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:09:06.100342 21988 webserver.cc:460] Webserver started at http://127.1.188.195:38447/ using document root <none> and password file <none>
I20260621 07:09:06.101269 21988 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:06.101393 21988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:06.101771 21988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:06.105208 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data/instance:
uuid: "d92671efd7884aedb4a195b68e09d1ce"
format_stamp: "Formatted at 2026-06-21 07:09:06 on dist-test-slave-hlq1"
server_key: "85da8a801b72ae24680eb2231b3ac462"
server_key_iv: "8f2df660787b2664daecc59899243c60"
server_key_version: "encryptionkey@0"
I20260621 07:09:06.106106 21988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance:
uuid: "d92671efd7884aedb4a195b68e09d1ce"
format_stamp: "Formatted at 2026-06-21 07:09:06 on dist-test-slave-hlq1"
server_key: "85da8a801b72ae24680eb2231b3ac462"
server_key_iv: "8f2df660787b2664daecc59899243c60"
server_key_version: "encryptionkey@0"
I20260621 07:09:06.112746 21988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260621 07:09:06.117359 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:06.119117 21988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260621 07:09:06.119341 21988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal
uuid: "d92671efd7884aedb4a195b68e09d1ce"
format_stamp: "Formatted at 2026-06-21 07:09:06 on dist-test-slave-hlq1"
server_key: "85da8a801b72ae24680eb2231b3ac462"
server_key_iv: "8f2df660787b2664daecc59899243c60"
server_key_version: "encryptionkey@0"
I20260621 07:09:06.119765 21988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:06.138175 21988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:06.139283 21988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:06.139676 21988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:06.140910 21988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 07:09:06.142736 21988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 07:09:06.142870 21988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:06.143015 21988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 07:09:06.143108 21988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:06.182994 21988 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.195:45439
I20260621 07:09:06.183070 22114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.195:45439 every 8 connection(s)
I20260621 07:09:06.184790 21988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/data/info.pb
I20260621 07:09:06.191471 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 21988
I20260621 07:09:06.191673 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal/instance
I20260621 07:09:06.192233 1779 external_mini_cluster.cc:1428] Setting key aff0a0aa3158840e422498093110ee48
I20260621 07:09:06.200013 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.187985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.195:55217 (local address 127.1.188.254:41233)
0621 07:09:06.188244 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.188249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.190004 (+ 1755us) server_negotiation.cc:407] Connection header received
0621 07:09:06.192331 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.192338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.192454 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.192585 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:06.193982 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.194830 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.196687 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.197142 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.197286 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.199028 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:06.199033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:06.199036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:06.199058 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:06.199099 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:06.199113 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:06.199273 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.199614 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.199723 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260621 07:09:06.201303 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.188655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41233 (local address 127.1.188.195:55217)
0621 07:09:06.189799 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.189840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.191999 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.192874 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.192884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.193003 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:06.193804 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.193823 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.194991 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.195003 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.196530 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.196543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.197407 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.197416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.197681 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.198665 (+ 984us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 07:09:06.198668 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:06.198717 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 07:09:06.198813 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:06.199294 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:06.199311 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:06.199319 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.199565 (+ 246us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.199962 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":344,"threads_started":1}
I20260621 07:09:06.203218 22115 heartbeater.cc:347] Connected to a master server at 127.1.188.254:41233
I20260621 07:09:06.203742 22115 heartbeater.cc:464] Registering TS with master...
I20260621 07:09:06.204998 22115 heartbeater.cc:511] Master 127.1.188.254:41233 requested a full tablet report, sending...
I20260621 07:09:06.207191 21667 ts_manager.cc:194] Registered new tserver with Master: d92671efd7884aedb4a195b68e09d1ce (127.1.188.195:45439)
I20260621 07:09:06.208155 1779 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 07:09:06.208688 21667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.188.195:55217
I20260621 07:09:06.220207 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.213548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.1.188.254:41233)
0621 07:09:06.213837 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.213849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.214224 (+ 375us) server_negotiation.cc:407] Connection header received
0621 07:09:06.214347 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.214352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.214436 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.214532 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:06.215035 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.215942 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.217479 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.217994 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.218129 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.219282 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:06.219288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:06.219291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:06.219312 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:06.219352 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:06.219366 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:06.219522 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.219690 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.219809 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 07:09:06.225281 21667 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 07:09:06.231789 21854 heartbeater.cc:503] Master 127.1.188.254:41233 was elected leader, sending a full tablet report...
W20260621 07:09:06.233599 21667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 07:09:06.260201 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.249118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.1.188.195:45439)
0621 07:09:06.250609 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.250621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.250661 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:09:06.250776 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.250783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.251017 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.251242 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:06.252462 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.254192 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.257374 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.258809 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.258987 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.259346 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.259581 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.259787 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":1098,"threads_started":1}
I20260621 07:09:06.260236 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.248755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:45439 (local address 127.0.0.1:50514)
0621 07:09:06.249760 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.249818 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.250012 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.251282 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.251291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.251339 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:06.251883 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.251900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.254376 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.254381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.257100 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.257114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.259031 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.259037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.259245 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.259304 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.259477 (+ 173us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.259715 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":38,"thread_start_us":2435,"threads_started":1}
I20260621 07:09:06.261457 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.248823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49750)
0621 07:09:06.251548 (+ 2725us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.251572 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.251694 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.252779 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.252784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.253089 (+ 305us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:06.253489 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.253503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.255482 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.255491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.257625 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.257637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.260013 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.260020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.260764 (+ 744us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.260803 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.260903 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.261059 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2545,"mutex_wait_us":66,"thread_start_us":468,"threads_started":1}
I20260621 07:09:06.261577 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.250042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.1.188.193:39431)
0621 07:09:06.250657 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.250665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.251837 (+ 1172us) server_negotiation.cc:407] Connection header received
0621 07:09:06.251997 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.252008 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.252280 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.252478 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:06.253691 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.255288 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.257854 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.259380 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.259572 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.259875 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.261000 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.261192 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":717,"threads_started":1}
I20260621 07:09:06.264410 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.251681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.194:40555 (local address 127.0.0.1:39862)
0621 07:09:06.252544 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.252564 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.252705 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.256658 (+ 3953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.256663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.256691 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:06.257241 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.257254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.259361 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.259366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.261713 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.261727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.263600 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.263607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.263773 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.263807 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.263898 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.264045 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":57,"thread_start_us":497,"threads_started":1}
I20260621 07:09:06.264626 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.252033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.1.188.194:40555)
0621 07:09:06.255991 (+ 3958us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.256002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.256034 (+ 32us) server_negotiation.cc:407] Connection header received
0621 07:09:06.256138 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.256147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.256422 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.256703 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:06.257487 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.259173 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.261966 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.263334 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.263542 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.263803 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.264038 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.264219 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3762,"thread_start_us":3616,"threads_started":1}
I20260621 07:09:06.265403 22050 tablet_service.cc:1467] Processing CreateTablet for tablet f75b355831f34d518bb0cfcd83796654 (DEFAULT_TABLE table=test-table [id=1bb209e577294070b2594c8738d163e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:06.266132 21789 tablet_service.cc:1467] Processing CreateTablet for tablet f75b355831f34d518bb0cfcd83796654 (DEFAULT_TABLE table=test-table [id=1bb209e577294070b2594c8738d163e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:06.267144 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75b355831f34d518bb0cfcd83796654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:06.267622 21789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75b355831f34d518bb0cfcd83796654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:06.269752 21920 tablet_service.cc:1467] Processing CreateTablet for tablet f75b355831f34d518bb0cfcd83796654 (DEFAULT_TABLE table=test-table [id=1bb209e577294070b2594c8738d163e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 07:09:06.271238 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75b355831f34d518bb0cfcd83796654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:06.280464 22133 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Bootstrap starting.
I20260621 07:09:06.283350 22134 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Bootstrap starting.
I20260621 07:09:06.285486 22133 tablet_bootstrap.cc:654] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:06.287161 22133 log.cc:824] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:06.287556 22134 tablet_bootstrap.cc:654] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:06.288954 22134 log.cc:824] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:06.291283 22133 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: No bootstrap required, opened a new log
I20260621 07:09:06.291749 22133 ts_tablet_manager.cc:1397] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260621 07:09:06.295670 22136 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Bootstrap starting.
I20260621 07:09:06.299908 22134 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: No bootstrap required, opened a new log
I20260621 07:09:06.300065 22136 tablet_bootstrap.cc:654] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:06.300299 22134 ts_tablet_manager.cc:1397] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260621 07:09:06.301585 22136 log.cc:824] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:06.304801 22133 raft_consensus.cc:348] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.305415 22133 raft_consensus.cc:374] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:06.305544 22133 raft_consensus.cc:729] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d30af9f5f8247a788087122a969e549, State: Initialized, Role: FOLLOWER
I20260621 07:09:06.306383 22133 consensus_queue.cc:260] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.308740 22133 ts_tablet_manager.cc:1428] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260621 07:09:06.312354 22136 tablet_bootstrap.cc:492] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: No bootstrap required, opened a new log
I20260621 07:09:06.312749 22136 ts_tablet_manager.cc:1397] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.005s
I20260621 07:09:06.313851 22134 raft_consensus.cc:348] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.314399 22134 raft_consensus.cc:374] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:06.314592 22134 raft_consensus.cc:729] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92671efd7884aedb4a195b68e09d1ce, State: Initialized, Role: FOLLOWER
I20260621 07:09:06.315294 22134 consensus_queue.cc:260] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.317269 22115 heartbeater.cc:503] Master 127.1.188.254:41233 was elected leader, sending a full tablet report...
I20260621 07:09:06.317870 22134 ts_tablet_manager.cc:1428] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260621 07:09:06.322975 22136 raft_consensus.cc:348] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.323488 22136 raft_consensus.cc:374] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:06.323637 22136 raft_consensus.cc:729] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165077c0cfb34d4589bb4db169ae964e, State: Initialized, Role: FOLLOWER
I20260621 07:09:06.324448 22136 consensus_queue.cc:260] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.325845 21985 heartbeater.cc:503] Master 127.1.188.254:41233 was elected leader, sending a full tablet report...
I20260621 07:09:06.326596 22141 raft_consensus.cc:482] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 07:09:06.326671 22136 ts_tablet_manager.cc:1428] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260621 07:09:06.326915 22141 raft_consensus.cc:504] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.329144 22141 leader_election.cc:288] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92671efd7884aedb4a195b68e09d1ce (127.1.188.195:45439), 3d30af9f5f8247a788087122a969e549 (127.1.188.193:39431)
I20260621 07:09:06.337718 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.329996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:55207 (local address 127.1.188.195:45439)
0621 07:09:06.330212 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.330219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.330384 (+ 165us) server_negotiation.cc:407] Connection header received
0621 07:09:06.330702 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.330709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.330811 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.330904 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:06.331597 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.332780 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.335583 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.336803 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.336981 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.337068 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.337285 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.337431 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:09:06.337878 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.329391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.195:45439 (local address 127.1.188.194:55207)
0621 07:09:06.330230 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.330260 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.330517 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.330923 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.330929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.330957 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:06.331422 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.331437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.332919 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.332925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.335358 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.335379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.336984 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.336989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.337125 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.337150 (+ 25us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.337225 (+ 75us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.337372 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":21,"thread_start_us":999,"threads_started":1}
I20260621 07:09:06.338171 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.329923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.188.194:54477 (local address 127.1.188.193:39431)
0621 07:09:06.330154 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.330162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.330820 (+ 658us) server_negotiation.cc:407] Connection header received
0621 07:09:06.331042 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.331047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.331147 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.331214 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 07:09:06.332284 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.333392 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.335890 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.337221 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.337394 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.337463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.337758 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.337906 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 07:09:06.338351 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.329435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.1.188.194:54477)
0621 07:09:06.330694 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.330710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:06.330840 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:06.331649 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:06.331654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:06.331675 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 07:09:06.332084 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.332105 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.333576 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.333582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.335722 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:06.335732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.337405 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.337410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:06.337574 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.337604 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 07:09:06.337703 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 07:09:06.337858 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":44,"thread_start_us":733,"threads_started":1}
I20260621 07:09:06.338986 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75b355831f34d518bb0cfcd83796654" candidate_uuid: "165077c0cfb34d4589bb4db169ae964e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92671efd7884aedb4a195b68e09d1ce" is_pre_election: true
I20260621 07:09:06.339371 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75b355831f34d518bb0cfcd83796654" candidate_uuid: "165077c0cfb34d4589bb4db169ae964e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d30af9f5f8247a788087122a969e549" is_pre_election: true
I20260621 07:09:06.339782 22070 raft_consensus.cc:2454] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 165077c0cfb34d4589bb4db169ae964e in term 0.
I20260621 07:09:06.340004 21809 raft_consensus.cc:2454] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 165077c0cfb34d4589bb4db169ae964e in term 0.
I20260621 07:09:06.340581 21876 leader_election.cc:302] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165077c0cfb34d4589bb4db169ae964e, d92671efd7884aedb4a195b68e09d1ce; no voters:
I20260621 07:09:06.341002 22141 raft_consensus.cc:2781] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 07:09:06.341161 22141 raft_consensus.cc:482] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 07:09:06.341282 22141 raft_consensus.cc:3037] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:06.343779 22141 raft_consensus.cc:504] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.344481 22141 leader_election.cc:288] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [CANDIDATE]: Term 1 election: Requested vote from peers d92671efd7884aedb4a195b68e09d1ce (127.1.188.195:45439), 3d30af9f5f8247a788087122a969e549 (127.1.188.193:39431)
I20260621 07:09:06.344986 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75b355831f34d518bb0cfcd83796654" candidate_uuid: "165077c0cfb34d4589bb4db169ae964e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92671efd7884aedb4a195b68e09d1ce"
I20260621 07:09:06.345046 21809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75b355831f34d518bb0cfcd83796654" candidate_uuid: "165077c0cfb34d4589bb4db169ae964e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d30af9f5f8247a788087122a969e549"
I20260621 07:09:06.345263 22070 raft_consensus.cc:3037] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:06.345281 21809 raft_consensus.cc:3037] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:06.348901 21809 raft_consensus.cc:2454] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165077c0cfb34d4589bb4db169ae964e in term 1.
I20260621 07:09:06.348901 22070 raft_consensus.cc:2454] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 165077c0cfb34d4589bb4db169ae964e in term 1.
I20260621 07:09:06.349342 21876 leader_election.cc:302] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 165077c0cfb34d4589bb4db169ae964e, d92671efd7884aedb4a195b68e09d1ce; no voters:
I20260621 07:09:06.349655 22141 raft_consensus.cc:2781] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:06.350185 22141 raft_consensus.cc:686] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 1 LEADER]: Becoming Leader. State: Replica: 165077c0cfb34d4589bb4db169ae964e, State: Running, Role: LEADER
I20260621 07:09:06.350718 22141 consensus_queue.cc:237] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } }
I20260621 07:09:06.356590 21668 catalog_manager.cc:5495] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e reported cstate change: term changed from 0 to 1, leader changed from <none> to 165077c0cfb34d4589bb4db169ae964e (127.1.188.194). New cstate: current_term: 1 leader_uuid: "165077c0cfb34d4589bb4db169ae964e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "165077c0cfb34d4589bb4db169ae964e" member_type: VOTER last_known_addr { host: "127.1.188.194" port: 40555 } health_report { overall_health: HEALTHY } } }
I20260621 07:09:06.389817 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.384257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.1.188.194:40555)
0621 07:09:06.384462 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.384467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.384488 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:09:06.384603 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.384607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.384710 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.384849 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:06.385187 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.385959 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.387527 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.387972 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.388076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.388508 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:06.389246 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:06.389382 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.389496 (+ 114us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.389569 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:09:06.399554 21809 raft_consensus.cc:1261] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 1 FOLLOWER]: Refusing update from remote peer 165077c0cfb34d4589bb4db169ae964e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:06.399554 22070 raft_consensus.cc:1261] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 1 FOLLOWER]: Refusing update from remote peer 165077c0cfb34d4589bb4db169ae964e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:06.400718 22144 consensus_queue.cc:1035] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92671efd7884aedb4a195b68e09d1ce" member_type: VOTER last_known_addr { host: "127.1.188.195" port: 45439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:06.401103 22141 consensus_queue.cc:1035] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d30af9f5f8247a788087122a969e549" member_type: VOTER last_known_addr { host: "127.1.188.193" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:06.413791 22147 mvcc.cc:205] Tried to move back new op lower bound from 7299177457240940544 to 7299177457057730560. Current Snapshot: MvccSnapshot[applied={T|T < 7299177457240940544}]
I20260621 07:09:06.415555 22148 mvcc.cc:205] Tried to move back new op lower bound from 7299177457240940544 to 7299177457057730560. Current Snapshot: MvccSnapshot[applied={T|T < 7299177457240940544}]
I20260621 07:09:06.429229 21667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:09:06.429620 21667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 07:09:06.433694 21667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ae8d116ac1444c5afdc5ac5ec91e66b: Sending DeleteTablet for 3 replicas of tablet f75b355831f34d518bb0cfcd83796654
I20260621 07:09:06.435122 21789 tablet_service.cc:1514] Processing DeleteTablet for tablet f75b355831f34d518bb0cfcd83796654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:06 UTC) from {username='slave'} at 127.0.0.1:49750
I20260621 07:09:06.435107 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet f75b355831f34d518bb0cfcd83796654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:06 UTC) from {username='slave'} at 127.0.0.1:50514
I20260621 07:09:06.435299 21920 tablet_service.cc:1514] Processing DeleteTablet for tablet f75b355831f34d518bb0cfcd83796654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 07:09:06 UTC) from {username='slave'} at 127.0.0.1:39862
I20260621 07:09:06.436203 22157 tablet_replica.cc:330] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: stopping tablet replica
I20260621 07:09:06.436764 22157 raft_consensus.cc:2229] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:09:06.437005 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21727
I20260621 07:09:06.437521 22157 raft_consensus.cc:2258] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:06.438460 22159 tablet_replica.cc:330] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: stopping tablet replica
I20260621 07:09:06.438956 22159 raft_consensus.cc:2229] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 07:09:06.439594 22158 tablet_replica.cc:330] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: stopping tablet replica
I20260621 07:09:06.439607 22159 raft_consensus.cc:2258] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:06.440011 22158 raft_consensus.cc:2229] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 1 LEADER]: Raft consensus shutting down.
I20260621 07:09:06.440794 22158 raft_consensus.cc:2258] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 07:09:06.442618 22157 ts_tablet_manager.cc:1905] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:09:06.444708 22158 ts_tablet_manager.cc:1905] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:09:06.446116 22159 ts_tablet_manager.cc:1905] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 07:09:06.448714 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.439825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.1.188.193:39431)
0621 07:09:06.440019 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.440025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.440046 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:09:06.440110 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.440115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.440225 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.440345 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:06.440998 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.442390 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.444323 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.445036 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.445147 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.446732 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:06.446745 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:06.446763 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:06.447666 (+ 903us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:06.447735 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:06.447761 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:06.447862 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.448126 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.448263 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 07:09:06.450214 21849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:06.457098 22158 ts_tablet_manager.cc:1918] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:06.457317 22158 log.cc:1196] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-1/wal/wals/f75b355831f34d518bb0cfcd83796654
I20260621 07:09:06.457949 22158 ts_tablet_manager.cc:1939] T f75b355831f34d518bb0cfcd83796654 P 165077c0cfb34d4589bb4db169ae964e: Deleting consensus metadata
I20260621 07:09:06.460261 21655 catalog_manager.cc:4841] TS 165077c0cfb34d4589bb4db169ae964e (127.1.188.194:40555): tablet f75b355831f34d518bb0cfcd83796654 (table test-table [id=1bb209e577294070b2594c8738d163e2]) successfully deleted
I20260621 07:09:06.461376 22159 ts_tablet_manager.cc:1918] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:06.461577 22159 log.cc:1196] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-2/wal/wals/f75b355831f34d518bb0cfcd83796654
I20260621 07:09:06.462095 22159 ts_tablet_manager.cc:1939] T f75b355831f34d518bb0cfcd83796654 P d92671efd7884aedb4a195b68e09d1ce: Deleting consensus metadata
I20260621 07:09:06.463860 21655 catalog_manager.cc:4841] TS d92671efd7884aedb4a195b68e09d1ce (127.1.188.195:45439): tablet f75b355831f34d518bb0cfcd83796654 (table test-table [id=1bb209e577294070b2594c8738d163e2]) successfully deleted
I20260621 07:09:06.565353 22157 ts_tablet_manager.cc:1918] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 07:09:06.565693 22157 log.cc:1196] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Deleting WAL directory at /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782025418728693-1779-0/minicluster-data/ts-0/wal/wals/f75b355831f34d518bb0cfcd83796654
I20260621 07:09:06.566259 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21727
I20260621 07:09:06.566404 22157 ts_tablet_manager.cc:1939] T f75b355831f34d518bb0cfcd83796654 P 3d30af9f5f8247a788087122a969e549: Deleting consensus metadata
I20260621 07:09:06.586942 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21858
W20260621 07:09:06.587016 21653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.188.193:39431 (error 108)
I20260621 07:09:06.589973 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.588874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49754)
0621 07:09:06.589194 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.589338 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
W20260621 07:09:06.590271 21653 catalog_manager.cc:4584] TS 3d30af9f5f8247a788087122a969e549 (127.1.188.193:39431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f75b355831f34d518bb0cfcd83796654: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
I20260621 07:09:06.596665 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.588355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.1.188.194:40555)
0621 07:09:06.588561 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.588569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.588594 (+ 25us) server_negotiation.cc:407] Connection header received
0621 07:09:06.588682 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.588688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.588813 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.588943 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:06.589555 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.590888 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.592464 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.593202 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.593336 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.594657 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:06.594671 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:06.594696 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:06.595738 (+ 1042us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:06.595833 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:06.595872 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:06.595983 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.596122 (+ 139us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.596238 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:09:06.597558 21980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:06.644533 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.643853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49756)
0621 07:09:06.644130 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.644271 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 07:09:06.645874 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.645434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49760)
0621 07:09:06.645611 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.645721 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 07:09:06.706655 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.705996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49776)
0621 07:09:06.706264 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.706390 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260621 07:09:06.707937 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.707533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49780)
0621 07:09:06.707695 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.707788 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 07:09:06.725900 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21858
I20260621 07:09:06.744431 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21988
I20260621 07:09:06.751865 22129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.745297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.1.188.195:45439)
0621 07:09:06.745500 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:06.745506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:06.745530 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:09:06.745723 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:06.745728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:06.745831 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:06.745925 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:06.746342 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.747149 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.748530 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:06.748948 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:06.749061 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:06.750216 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:06.750226 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:06.750244 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:06.750938 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:06.751010 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:06.751034 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:06.751124 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:06.751281 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 07:09:06.751387 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 07:09:06.752604 22110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:06.789520 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.788864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49782)
0621 07:09:06.789151 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.789284 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 07:09:06.790933 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:06.790399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.193:39431 (local address 127.0.0.1:49794)
0621 07:09:06.790689 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:06.790782 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.193:39431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260621 07:09:06.848309 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21988
I20260621 07:09:06.866154 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 21637
I20260621 07:09:06.867475 21698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:06.979315 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 21637
2026-06-21T07:09:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2825 ms)
[----------] 6 tests from EncryptionPolicyTest (22688 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:07 dist-test-slave-hlq1 krb5kdc[22168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:07 dist-test-slave-hlq1 krb5kdc[22168](info): set up 2 sockets
Jun 21 07:09:07 dist-test-slave-hlq1 krb5kdc[22168](info): commencing operation
krb5kdc: starting...
W20260621 07:09:08.108090 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:08 dist-test-slave-hlq1 krb5kdc[22168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:08.355497 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:08.700407 22181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:08.700721 22181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:08.700891 22181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:08.708062 22181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:08.708163 22181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:08.708268 22181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:08.708412 22181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:08.708501 22181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:08.718240 22181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:08.719569 22181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:08.720857 22181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:08.732949 22189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:08.732990 22186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:08.733033 22187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:08.734329 22181 server_base.cc:1034] running on GCE node
I20260621 07:09:08.735088 22181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:08.736526 22181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:08.737761 22181 hybrid_clock.cc:648] HybridClock initialized: now 1782025748737720 us; error 53 us; skew 500 ppm
I20260621 07:09:08.738138 22181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:08 dist-test-slave-hlq1 krb5kdc[22168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:08.741277 22181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:08.742926 22181 webserver.cc:460] Webserver started at http://127.0.0.1:41793/ using document root <none> and password file <none>
I20260621 07:09:08.743717 22181 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:08.743866 22181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:08.744174 22181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:08.747149 22181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "8e946cc2a7394251ae9fd0b282acb474"
format_stamp: "Formatted at 2026-06-21 07:09:08 on dist-test-slave-hlq1"
server_key: "821999cdbe8411399f6ab6473d8f9724"
server_key_iv: "5af521fd1a7a9f7f91ded2d374c3b6be"
server_key_version: "encryptionkey@0"
I20260621 07:09:08.747972 22181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "8e946cc2a7394251ae9fd0b282acb474"
format_stamp: "Formatted at 2026-06-21 07:09:08 on dist-test-slave-hlq1"
server_key: "821999cdbe8411399f6ab6473d8f9724"
server_key_iv: "5af521fd1a7a9f7f91ded2d374c3b6be"
server_key_version: "encryptionkey@0"
I20260621 07:09:08.753080 22181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 07:09:08.756608 22196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:08.757999 22181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:08.758203 22181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "8e946cc2a7394251ae9fd0b282acb474"
format_stamp: "Formatted at 2026-06-21 07:09:08 on dist-test-slave-hlq1"
server_key: "821999cdbe8411399f6ab6473d8f9724"
server_key_iv: "5af521fd1a7a9f7f91ded2d374c3b6be"
server_key_version: "encryptionkey@0"
I20260621 07:09:08.758423 22181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:08.772559 22181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:08.777196 22181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:08.777508 22181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:08.797307 22181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44837
I20260621 07:09:08.797392 22247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44837 every 8 connection(s)
I20260621 07:09:08.798719 22181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:08.799258 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22181
I20260621 07:09:08.799492 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:08.800364 1779 external_mini_cluster.cc:1428] Setting key a833b3e794ae3b13b5409c6d17a5bd0e
I20260621 07:09:08.803575 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 07:09:08 dist-test-slave-hlq1 krb5kdc[22168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:09:08.817251 22248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474: Bootstrap starting.
I20260621 07:09:08.821425 22248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:08.822804 22248 log.cc:824] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:08.825739 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:08.802429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56021 (local address 127.0.0.1:44837)
0621 07:09:08.803328 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:08.803340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:08.803380 (+ 40us) server_negotiation.cc:407] Connection header received
0621 07:09:08.805194 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:08.805216 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:08.805577 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:08.805884 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:08.807733 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:08.809115 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:08.810965 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:08.811747 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:08.811880 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:08.816802 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:08.816816 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:08.816836 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:08.816888 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:08.820852 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:08.821765 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:08.821775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:08.821784 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:08.822017 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:08.822500 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:08.822507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:08.822513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:08.823192 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:08.823592 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:08.824062 (+ 470us) server_negotiation.cc:299] Negotiation successful
0621 07:09:08.824431 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":456,"threads_started":1}
I20260621 07:09:08.826388 22248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474: No bootstrap required, opened a new log
I20260621 07:09:08.829985 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:08.834627 22248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260621 07:09:08.835093 22248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:08.835217 22248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e946cc2a7394251ae9fd0b282acb474, State: Initialized, Role: FOLLOWER
I20260621 07:09:08.835912 22248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260621 07:09:08.836151 22248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:08.836316 22248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:08.836493 22248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:08.838634 22248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260621 07:09:08.839125 22248 leader_election.cc:302] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e946cc2a7394251ae9fd0b282acb474; no voters:
I20260621 07:09:08.839830 22248 leader_election.cc:288] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:08.839964 22258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:08.842402 22248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:08.843645 22258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [term 1 LEADER]: Becoming Leader. State: Replica: 8e946cc2a7394251ae9fd0b282acb474, State: Running, Role: LEADER
I20260621 07:09:08.844332 22258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260621 07:09:08.847457 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e946cc2a7394251ae9fd0b282acb474. Latest consensus state: current_term: 1 leader_uuid: "8e946cc2a7394251ae9fd0b282acb474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } } }
I20260621 07:09:08.847868 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:08.847832 22259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e946cc2a7394251ae9fd0b282acb474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e946cc2a7394251ae9fd0b282acb474" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } } }
I20260621 07:09:08.848131 22259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:08.854471 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:08.834779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37713 (local address 127.0.0.1:44837)
0621 07:09:08.835009 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:08.835018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:08.835578 (+ 560us) server_negotiation.cc:407] Connection header received
0621 07:09:08.835773 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:08.835783 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:08.835841 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:08.835944 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:08.837575 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:08.838332 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:08.839778 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:08.840558 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:08.840688 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:08.843516 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:08.843522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:08.843527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:08.843566 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:08.848917 (+ 5351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:08.850244 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:08.850252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:08.850258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:08.850482 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:08.851209 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:08.851217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:08.851222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:08.851732 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:08.853458 (+ 1726us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:08.853691 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 07:09:08.853925 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":64}
I20260621 07:09:08.863500 22269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:08.868104 22269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:08.876673 22269 catalog_manager.cc:1347] Generated new cluster ID: f81f7231a7a144048c9d9f74b0562025
I20260621 07:09:08.876806 22269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:08.889374 22269 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:08.890892 22269 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:08.896767 22269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e946cc2a7394251ae9fd0b282acb474: Generated new TSK 0
I20260621 07:09:08.897584 22269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:08.904230 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22181
I20260621 07:09:08.905678 22243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:08.995031 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22181
W20260621 07:09:09.012396 22166 connection.cc:535] client connection to 127.0.0.1:44837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44837): Connection reset by peer (error 104)
2026-06-21T07:09:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:09 dist-test-slave-hlq1 krb5kdc[22280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:09 dist-test-slave-hlq1 krb5kdc[22280](info): set up 2 sockets
Jun 21 07:09:09 dist-test-slave-hlq1 krb5kdc[22280](info): commencing operation
krb5kdc: starting...
W20260621 07:09:10.107838 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:10 dist-test-slave-hlq1 krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:10.354939 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:10.686596 22293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:10.686967 22293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:10.687129 22293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:10.694428 22293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:10.694583 22293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:10.694700 22293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:10.694811 22293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:10.694891 22293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:10.705083 22293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:10.706547 22293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:10.707954 22293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:10.719856 22298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:10.719795 22299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:10.719866 22301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:10.721176 22293 server_base.cc:1034] running on GCE node
I20260621 07:09:10.721891 22293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:10.723218 22293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:10.724431 22293 hybrid_clock.cc:648] HybridClock initialized: now 1782025750724395 us; error 57 us; skew 500 ppm
I20260621 07:09:10.724732 22293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:10 dist-test-slave-hlq1 krb5kdc[22280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:10.730715 22293 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:10.732376 22293 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
I20260621 07:09:10.733027 22293 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:10.733132 22293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:10.733392 22293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:10.736364 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "f3bf4338c2ce49518654613a7a0f4008"
format_stamp: "Formatted at 2026-06-21 07:09:10 on dist-test-slave-hlq1"
server_key: "539fac53dda1d40a415448de15678f7f"
server_key_iv: "561faed8b350db5d08597146ed5146bf"
server_key_version: "encryptionkey@0"
I20260621 07:09:10.737073 22293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "f3bf4338c2ce49518654613a7a0f4008"
format_stamp: "Formatted at 2026-06-21 07:09:10 on dist-test-slave-hlq1"
server_key: "539fac53dda1d40a415448de15678f7f"
server_key_iv: "561faed8b350db5d08597146ed5146bf"
server_key_version: "encryptionkey@0"
I20260621 07:09:10.742278 22293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:09:10.745836 22308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:10.747195 22293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260621 07:09:10.747393 22293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "f3bf4338c2ce49518654613a7a0f4008"
format_stamp: "Formatted at 2026-06-21 07:09:10 on dist-test-slave-hlq1"
server_key: "539fac53dda1d40a415448de15678f7f"
server_key_iv: "561faed8b350db5d08597146ed5146bf"
server_key_version: "encryptionkey@0"
I20260621 07:09:10.747678 22293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:10.761512 22293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:10.765552 22293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:10.765841 22293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:10.784715 22293 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37551
I20260621 07:09:10.784759 22359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37551 every 8 connection(s)
I20260621 07:09:10.786075 22293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:10.788666 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22293
I20260621 07:09:10.788849 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:10.789268 1779 external_mini_cluster.cc:1428] Setting key 79b58679f78bfe206b7e62f43f4da555
I20260621 07:09:10.789827 22360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:10.802345 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:10.791038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38117 (local address 127.0.0.1:37551)
0621 07:09:10.791847 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:10.791858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:10.791895 (+ 37us) server_negotiation.cc:407] Connection header received
0621 07:09:10.793387 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:10.793405 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:10.793784 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:10.794173 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:10.794699 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:10.796068 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:10.797350 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:10.797985 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:10.798169 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:10.799633 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:10.799644 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:10.799666 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:10.800397 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:10.800487 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:10.800528 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:10.800696 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:10.800963 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 07:09:10.801137 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":399,"threads_started":1}
I20260621 07:09:10.803261 22360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008: Bootstrap starting.
I20260621 07:09:10.806174 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:10.806599 22360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:10.807627 22360 log.cc:824] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:10.810886 22360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008: No bootstrap required, opened a new log
I20260621 07:09:10.817719 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:10.810737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58709 (local address 127.0.0.1:37551)
0621 07:09:10.810923 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:10.810931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:10.811442 (+ 511us) server_negotiation.cc:407] Connection header received
0621 07:09:10.811642 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:10.811648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:10.811723 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:10.811836 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:10.812341 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:10.813534 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:10.814843 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:10.815542 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:10.815684 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:10.816891 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:10.816901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:10.816906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:10.816928 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:10.816976 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:10.816996 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:10.817118 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:10.817306 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 07:09:10.817365 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 07:09:10.820499 22360 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260621 07:09:10.820870 22360 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:10.820986 22360 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3bf4338c2ce49518654613a7a0f4008, State: Initialized, Role: FOLLOWER
I20260621 07:09:10.821583 22360 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260621 07:09:10.821797 22360 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:10.821962 22360 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:10.822135 22360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:10.824337 22360 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260621 07:09:10.824882 22360 leader_election.cc:302] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3bf4338c2ce49518654613a7a0f4008; no voters:
I20260621 07:09:10.825639 22360 leader_election.cc:288] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:10.825819 22370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:10.826591 22370 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [term 1 LEADER]: Becoming Leader. State: Replica: f3bf4338c2ce49518654613a7a0f4008, State: Running, Role: LEADER
I20260621 07:09:10.827114 22370 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260621 07:09:10.827800 22360 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:10.830734 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3bf4338c2ce49518654613a7a0f4008. Latest consensus state: current_term: 1 leader_uuid: "f3bf4338c2ce49518654613a7a0f4008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260621 07:09:10.830686 22371 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3bf4338c2ce49518654613a7a0f4008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3bf4338c2ce49518654613a7a0f4008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260621 07:09:10.831122 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:10.831182 22371 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:10.832820 22379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:10.837513 22379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:10.845639 22379 catalog_manager.cc:1347] Generated new cluster ID: 25db54ba327c4b0fb5f596149b9c95ae
I20260621 07:09:10.845760 22379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:10.859618 22379 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:10.861433 22379 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:10.869526 22379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3bf4338c2ce49518654613a7a0f4008: Generated new TSK 0
I20260621 07:09:10.870482 22379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:10.879082 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22293
W20260621 07:09:10.880489 22355 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38117
W20260621 07:09:10.881325 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22293: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:10.881565 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22293
W20260621 07:09:10.898530 22278 connection.cc:535] client connection to 127.0.0.1:37551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37551): Connection reset by peer (error 104)
2026-06-21T07:09:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:10 dist-test-slave-hlq1 krb5kdc[22391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:10 dist-test-slave-hlq1 krb5kdc[22391](info): set up 2 sockets
Jun 21 07:09:10 dist-test-slave-hlq1 krb5kdc[22391](info): commencing operation
krb5kdc: starting...
W20260621 07:09:11.995939 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:12 dist-test-slave-hlq1 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:12.259078 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:12.587776 22404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:12.588070 22404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:12.588168 22404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:12.595351 22404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:12.595474 22404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:12.595523 22404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:12.595584 22404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:12.595628 22404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:12.605582 22404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:12.606841 22404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:12.608224 22404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:12.620133 22409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:12.620083 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:12.620209 22412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:12.621402 22404 server_base.cc:1034] running on GCE node
I20260621 07:09:12.622267 22404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:12.623802 22404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:12.625062 22404 hybrid_clock.cc:648] HybridClock initialized: now 1782025752625033 us; error 43 us; skew 500 ppm
I20260621 07:09:12.625442 22404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:12 dist-test-slave-hlq1 krb5kdc[22391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:12.628770 22404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:12.630389 22404 webserver.cc:460] Webserver started at http://127.0.0.1:37525/ using document root <none> and password file <none>
I20260621 07:09:12.631096 22404 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:12.631249 22404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:12.631742 22404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:12.634827 22404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "431aeff3ff214885a6eec1422de42483"
format_stamp: "Formatted at 2026-06-21 07:09:12 on dist-test-slave-hlq1"
server_key: "a2b02df03bbfe1acd3be802c8390b63c"
server_key_iv: "343edce2c718a42744c83c88e80a1801"
server_key_version: "encryptionkey@0"
I20260621 07:09:12.635673 22404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "431aeff3ff214885a6eec1422de42483"
format_stamp: "Formatted at 2026-06-21 07:09:12 on dist-test-slave-hlq1"
server_key: "a2b02df03bbfe1acd3be802c8390b63c"
server_key_iv: "343edce2c718a42744c83c88e80a1801"
server_key_version: "encryptionkey@0"
I20260621 07:09:12.640955 22404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260621 07:09:12.644562 22419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:12.645947 22404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 07:09:12.646130 22404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "431aeff3ff214885a6eec1422de42483"
format_stamp: "Formatted at 2026-06-21 07:09:12 on dist-test-slave-hlq1"
server_key: "a2b02df03bbfe1acd3be802c8390b63c"
server_key_iv: "343edce2c718a42744c83c88e80a1801"
server_key_version: "encryptionkey@0"
I20260621 07:09:12.646341 22404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:12.652058 22404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:12.656423 22404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:12.656725 22404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:12.675158 22404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43321
I20260621 07:09:12.675204 22470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43321 every 8 connection(s)
I20260621 07:09:12.676618 22404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:12.680317 22471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:12.683965 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22404
I20260621 07:09:12.684156 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:12.684703 1779 external_mini_cluster.cc:1428] Setting key 889a07da1195cb86f994aa06a9ba9c16
I20260621 07:09:12.692430 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:12.686539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37531 (local address 127.0.0.1:43321)
0621 07:09:12.687356 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:12.687368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:12.687455 (+ 87us) server_negotiation.cc:407] Connection header received
0621 07:09:12.689102 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:12.689123 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:12.689457 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:12.689789 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:12.690846 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:12.690851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:12.690863 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:12.690884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:12.690942 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:12.690962 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:12.691134 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:12.691265 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 07:09:12.691498 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":375,"threads_started":1}
I20260621 07:09:12.693095 22471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483: Bootstrap starting.
I20260621 07:09:12.696076 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:12.696338 22471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:12.697602 22471 log.cc:824] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:12.701064 22471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483: No bootstrap required, opened a new log
I20260621 07:09:12.704025 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:12.701424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55813 (local address 127.0.0.1:43321)
0621 07:09:12.701653 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:12.701660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:12.701942 (+ 282us) server_negotiation.cc:407] Connection header received
0621 07:09:12.702145 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:12.702157 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:12.702207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:12.702364 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:12.703311 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:12.703315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:12.703318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:12.703333 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:12.703370 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:12.703385 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:12.703572 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:12.703662 (+ 90us) server_negotiation.cc:299] Negotiation successful
0621 07:09:12.703727 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:09:12.709704 22471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } }
I20260621 07:09:12.710206 22471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:12.710331 22471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431aeff3ff214885a6eec1422de42483, State: Initialized, Role: FOLLOWER
I20260621 07:09:12.710916 22471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } }
I20260621 07:09:12.711148 22471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:12.711321 22471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:12.711515 22471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:12.714067 22471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } }
I20260621 07:09:12.714612 22471 leader_election.cc:302] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431aeff3ff214885a6eec1422de42483; no voters:
I20260621 07:09:12.715186 22471 leader_election.cc:288] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:12.715350 22481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:12.716059 22481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [term 1 LEADER]: Becoming Leader. State: Replica: 431aeff3ff214885a6eec1422de42483, State: Running, Role: LEADER
I20260621 07:09:12.716665 22481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } }
I20260621 07:09:12.717321 22471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:12.719691 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431aeff3ff214885a6eec1422de42483. Latest consensus state: current_term: 1 leader_uuid: "431aeff3ff214885a6eec1422de42483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } } }
I20260621 07:09:12.719933 22482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431aeff3ff214885a6eec1422de42483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431aeff3ff214885a6eec1422de42483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43321 } } }
I20260621 07:09:12.720181 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:12.720229 22482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:12.721817 22489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:12.727231 22489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:12.735589 22489 catalog_manager.cc:1347] Generated new cluster ID: 5c63775d739a433488784e69001a5947
I20260621 07:09:12.735713 22489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:12.755357 22489 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:12.755846 22489 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:12.763968 22489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431aeff3ff214885a6eec1422de42483: Generated new TSK 0
I20260621 07:09:12.764811 22489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:12.781180 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22404
W20260621 07:09:12.782383 22466 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37531
W20260621 07:09:12.783136 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22404: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:12.783295 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22404
2026-06-21T07:09:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:12 dist-test-slave-hlq1 krb5kdc[22502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:12 dist-test-slave-hlq1 krb5kdc[22502](info): set up 2 sockets
Jun 21 07:09:12 dist-test-slave-hlq1 krb5kdc[22502](info): commencing operation
krb5kdc: starting...
W20260621 07:09:13.899899 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:14 dist-test-slave-hlq1 krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:14.195292 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:14.579730 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:14.580157 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:14.580335 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:14.588368 22515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:14.588541 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:14.588655 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:14.588768 22515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:14.588850 22515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:14.600085 22515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:14.601828 22515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:14.603510 22515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:14.617137 22520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:14.617111 22521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:14.617236 22523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:14.618212 22515 server_base.cc:1034] running on GCE node
I20260621 07:09:14.619117 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:14.620945 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:14.622186 22515 hybrid_clock.cc:648] HybridClock initialized: now 1782025754622144 us; error 39 us; skew 500 ppm
I20260621 07:09:14.622592 22515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:14 dist-test-slave-hlq1 krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:14.626142 22515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:14.627842 22515 webserver.cc:460] Webserver started at http://127.0.0.1:40099/ using document root <none> and password file <none>
I20260621 07:09:14.628635 22515 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:14.628736 22515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:14.629019 22515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:14.632488 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "b4380171a78b4727934d194fed245eba"
format_stamp: "Formatted at 2026-06-21 07:09:14 on dist-test-slave-hlq1"
server_key: "3e874c639fde071420bb75328784cd91"
server_key_iv: "b586f9aba84ae1ca8239958cd1d93d0b"
server_key_version: "encryptionkey@0"
I20260621 07:09:14.633296 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "b4380171a78b4727934d194fed245eba"
format_stamp: "Formatted at 2026-06-21 07:09:14 on dist-test-slave-hlq1"
server_key: "3e874c639fde071420bb75328784cd91"
server_key_iv: "b586f9aba84ae1ca8239958cd1d93d0b"
server_key_version: "encryptionkey@0"
I20260621 07:09:14.639156 22515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:09:14.643092 22530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:14.644666 22515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:09:14.644829 22515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "b4380171a78b4727934d194fed245eba"
format_stamp: "Formatted at 2026-06-21 07:09:14 on dist-test-slave-hlq1"
server_key: "3e874c639fde071420bb75328784cd91"
server_key_iv: "b586f9aba84ae1ca8239958cd1d93d0b"
server_key_version: "encryptionkey@0"
I20260621 07:09:14.645013 22515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:14.673277 22515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:14.678489 22515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:14.678787 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:14.700590 22515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44549
I20260621 07:09:14.700646 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44549 every 8 connection(s)
I20260621 07:09:14.702212 22515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:14.706697 22582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:14.707710 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22515
I20260621 07:09:14.707926 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:14.708560 1779 external_mini_cluster.cc:1428] Setting key 14ad6649b5f42d3e0a915f18adaee7bb
I20260621 07:09:14.720710 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba: Bootstrap starting.
Jun 21 07:09:14 dist-test-slave-hlq1 krb5kdc[22502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:09:14.724409 22582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:14.725648 22582 log.cc:824] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:14.728961 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba: No bootstrap required, opened a new log
I20260621 07:09:14.732728 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:14.710747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41993 (local address 127.0.0.1:44549)
0621 07:09:14.711693 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:14.711706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:14.711753 (+ 47us) server_negotiation.cc:407] Connection header received
0621 07:09:14.713928 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:14.713953 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:14.714401 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:14.714838 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:14.716507 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:14.717946 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:14.719423 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:14.720205 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:14.720413 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:14.724883 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:14.724892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:14.724907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:14.724956 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:14.728300 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:14.729025 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:14.729039 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:14.729047 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:14.729272 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:14.729690 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:14.729696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:14.729700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:14.730259 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:14.730616 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:14.731114 (+ 498us) server_negotiation.cc:299] Negotiation successful
0621 07:09:14.731621 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":378,"threads_started":1}
I20260621 07:09:14.736975 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:14.738286 22582 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260621 07:09:14.738694 22582 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:14.738811 22582 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4380171a78b4727934d194fed245eba, State: Initialized, Role: FOLLOWER
I20260621 07:09:14.739595 22582 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260621 07:09:14.739861 22582 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:14.740025 22582 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:14.740221 22582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:14.742977 22582 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260621 07:09:14.743667 22582 leader_election.cc:302] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4380171a78b4727934d194fed245eba; no voters:
I20260621 07:09:14.744396 22582 leader_election.cc:288] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:14.744508 22592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:14.747290 22582 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:14.747745 22592 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [term 1 LEADER]: Becoming Leader. State: Replica: b4380171a78b4727934d194fed245eba, State: Running, Role: LEADER
I20260621 07:09:14.748420 22592 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } }
I20260621 07:09:14.752686 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4380171a78b4727934d194fed245eba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } } }
I20260621 07:09:14.752849 22594 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4380171a78b4727934d194fed245eba. Latest consensus state: current_term: 1 leader_uuid: "b4380171a78b4727934d194fed245eba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4380171a78b4727934d194fed245eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44549 } } }
I20260621 07:09:14.753201 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:14.753201 22594 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:14.754216 22600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:14.762316 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:14.742096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38841 (local address 127.0.0.1:44549)
0621 07:09:14.742297 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:14.742304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:14.742732 (+ 428us) server_negotiation.cc:407] Connection header received
0621 07:09:14.742860 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:14.742866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:14.742966 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:14.743089 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:14.744910 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:14.745957 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:14.747354 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:14.748121 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:14.748270 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:14.751965 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:14.751972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:14.751979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:14.752029 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:14.756728 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:14.757979 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:14.757988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:14.757994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:14.758205 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:14.758781 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:14.758789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:14.758795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:14.759477 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:14.759828 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:14.761376 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0621 07:09:14.761708 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:09:14.768079 22600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:14.781940 22600 catalog_manager.cc:1347] Generated new cluster ID: 035459781b6e469eacde803103c6b0a4
I20260621 07:09:14.782119 22600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:14.798084 22600 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:14.799744 22600 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:14.813019 22600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4380171a78b4727934d194fed245eba: Generated new TSK 0
I20260621 07:09:14.814011 22600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:14.820783 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22515
I20260621 07:09:14.822124 22577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:14.924453 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22515
2026-06-21T07:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:15 dist-test-slave-hlq1 krb5kdc[22614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:15 dist-test-slave-hlq1 krb5kdc[22614](info): set up 2 sockets
Jun 21 07:09:15 dist-test-slave-hlq1 krb5kdc[22614](info): commencing operation
krb5kdc: starting...
W20260621 07:09:16.060374 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.007s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:16 dist-test-slave-hlq1 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:16.371330 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:16.759315 22627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:16.759799 22627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:16.759908 22627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:16.767863 22627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:16.768002 22627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:16.768055 22627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:16.768126 22627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:16.768175 22627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:16.779155 22627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:16.781229 22627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:16.782800 22627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:16.796628 22632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:16.796657 22633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:16.796819 22635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:16.797628 22627 server_base.cc:1034] running on GCE node
I20260621 07:09:16.798640 22627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:16.800487 22627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:16.801719 22627 hybrid_clock.cc:648] HybridClock initialized: now 1782025756801666 us; error 39 us; skew 500 ppm
I20260621 07:09:16.802150 22627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:16 dist-test-slave-hlq1 krb5kdc[22614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:16.811285 22627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:16.813205 22627 webserver.cc:460] Webserver started at http://127.0.0.1:35529/ using document root <none> and password file <none>
I20260621 07:09:16.814042 22627 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:16.814152 22627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:16.814455 22627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:16.818234 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "8744aa6a291946e3bc2fb8c4f1ca347e"
format_stamp: "Formatted at 2026-06-21 07:09:16 on dist-test-slave-hlq1"
server_key: "49f7c33fe07e61aa59087ac787911c23"
server_key_iv: "c775dc967065b12b54992f8ac5117aee"
server_key_version: "encryptionkey@0"
I20260621 07:09:16.819082 22627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "8744aa6a291946e3bc2fb8c4f1ca347e"
format_stamp: "Formatted at 2026-06-21 07:09:16 on dist-test-slave-hlq1"
server_key: "49f7c33fe07e61aa59087ac787911c23"
server_key_iv: "c775dc967065b12b54992f8ac5117aee"
server_key_version: "encryptionkey@0"
I20260621 07:09:16.825182 22627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260621 07:09:16.829248 22642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:16.830828 22627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:09:16.831043 22627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "8744aa6a291946e3bc2fb8c4f1ca347e"
format_stamp: "Formatted at 2026-06-21 07:09:16 on dist-test-slave-hlq1"
server_key: "49f7c33fe07e61aa59087ac787911c23"
server_key_iv: "c775dc967065b12b54992f8ac5117aee"
server_key_version: "encryptionkey@0"
I20260621 07:09:16.831308 22627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:16.844396 22627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:16.849099 22627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:16.849432 22627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:16.870635 22627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37443
I20260621 07:09:16.870667 22693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37443 every 8 connection(s)
I20260621 07:09:16.872435 22627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:16.872991 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22627
I20260621 07:09:16.873208 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:16.873934 1779 external_mini_cluster.cc:1428] Setting key 63dde915ca544b80732250edadbb3609
I20260621 07:09:16.878077 22694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:16.890985 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:16.876147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43851 (local address 127.0.0.1:37443)
0621 07:09:16.877036 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:16.877053 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:16.877114 (+ 61us) server_negotiation.cc:407] Connection header received
0621 07:09:16.879692 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:16.879724 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:16.880303 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:16.880791 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:16.881348 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:16.883427 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:16.884904 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:16.886019 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:16.886195 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:16.887625 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:16.887635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:16.887664 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:16.888475 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:16.888605 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:16.888657 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:16.888786 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:16.889059 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 07:09:16.889391 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":361,"threads_started":1}
I20260621 07:09:16.893743 22694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e: Bootstrap starting.
I20260621 07:09:16.895515 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:16.897624 22694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:16.898892 22694 log.cc:824] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:16.902734 22694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e: No bootstrap required, opened a new log
I20260621 07:09:16.907915 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:16.900576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56017 (local address 127.0.0.1:37443)
0621 07:09:16.900819 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:16.900829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:16.901226 (+ 397us) server_negotiation.cc:407] Connection header received
0621 07:09:16.901451 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:16.901455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:16.901527 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:16.901624 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:16.902321 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:16.903194 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:16.905134 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:16.905617 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:16.905712 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:16.906999 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:16.907003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:16.907007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:16.907029 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:16.907072 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:16.907089 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:16.907182 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:16.907328 (+ 146us) server_negotiation.cc:299] Negotiation successful
0621 07:09:16.907476 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:09:16.913935 22694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260621 07:09:16.914484 22694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:16.914603 22694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8744aa6a291946e3bc2fb8c4f1ca347e, State: Initialized, Role: FOLLOWER
I20260621 07:09:16.915438 22694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260621 07:09:16.915827 22694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:16.916002 22694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:16.916177 22694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:16.918649 22694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260621 07:09:16.919498 22694 leader_election.cc:302] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8744aa6a291946e3bc2fb8c4f1ca347e; no voters:
I20260621 07:09:16.920154 22694 leader_election.cc:288] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:16.920344 22704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:16.921307 22704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [term 1 LEADER]: Becoming Leader. State: Replica: 8744aa6a291946e3bc2fb8c4f1ca347e, State: Running, Role: LEADER
I20260621 07:09:16.921905 22704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } }
I20260621 07:09:16.922863 22694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:16.925750 22706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8744aa6a291946e3bc2fb8c4f1ca347e. Latest consensus state: current_term: 1 leader_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260621 07:09:16.926244 22706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:16.926107 22705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8744aa6a291946e3bc2fb8c4f1ca347e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37443 } } }
I20260621 07:09:16.926379 22705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:16.927204 22713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:16.933725 22713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:16.945843 22713 catalog_manager.cc:1347] Generated new cluster ID: 18f48495f364472ebcdfbd9452b7f601
I20260621 07:09:16.946002 22713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:16.977515 22713 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:16.979645 22713 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:16.986495 22713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8744aa6a291946e3bc2fb8c4f1ca347e: Generated new TSK 0
I20260621 07:09:16.987787 22713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:16.997009 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22627
W20260621 07:09:16.998466 22689 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43851
W20260621 07:09:16.999333 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22627: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:16.999517 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22627
2026-06-21T07:09:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:17 dist-test-slave-hlq1 krb5kdc[22725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:17 dist-test-slave-hlq1 krb5kdc[22725](info): set up 2 sockets
Jun 21 07:09:17 dist-test-slave-hlq1 krb5kdc[22725](info): commencing operation
krb5kdc: starting...
W20260621 07:09:18.140686 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:18 dist-test-slave-hlq1 krb5kdc[22725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:18.399006 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:18.755563 22738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:18.755851 22738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:18.756004 22738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:18.763309 22738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:18.763468 22738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:18.763574 22738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:18.763657 22738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:18.763734 22738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:18.773910 22738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:18.775270 22738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:18.776628 22738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:18.788913 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:18.789005 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:18.788985 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:18.789757 22738 server_base.cc:1034] running on GCE node
I20260621 07:09:18.790544 22738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:18.792143 22738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:18.793416 22738 hybrid_clock.cc:648] HybridClock initialized: now 1782025758793357 us; error 38 us; skew 500 ppm
I20260621 07:09:18.793843 22738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:18 dist-test-slave-hlq1 krb5kdc[22725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:18.797516 22738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:18.799238 22738 webserver.cc:460] Webserver started at http://127.0.0.1:42019/ using document root <none> and password file <none>
I20260621 07:09:18.800107 22738 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:18.800287 22738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:18.800650 22738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:18.804394 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0"
format_stamp: "Formatted at 2026-06-21 07:09:18 on dist-test-slave-hlq1"
server_key: "dfb95847876c3c0bf928dd5548de924e"
server_key_iv: "18497a03050205875143f8706dd52960"
server_key_version: "encryptionkey@0"
I20260621 07:09:18.805517 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0"
format_stamp: "Formatted at 2026-06-21 07:09:18 on dist-test-slave-hlq1"
server_key: "dfb95847876c3c0bf928dd5548de924e"
server_key_iv: "18497a03050205875143f8706dd52960"
server_key_version: "encryptionkey@0"
I20260621 07:09:18.811372 22738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 07:09:18.815614 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:18.817157 22738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 07:09:18.817404 22738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0"
format_stamp: "Formatted at 2026-06-21 07:09:18 on dist-test-slave-hlq1"
server_key: "dfb95847876c3c0bf928dd5548de924e"
server_key_iv: "18497a03050205875143f8706dd52960"
server_key_version: "encryptionkey@0"
I20260621 07:09:18.817618 22738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:18.824337 22738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:18.828795 22738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:18.829135 22738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:18.850066 22738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36097
I20260621 07:09:18.850108 22804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36097 every 8 connection(s)
I20260621 07:09:18.851682 22738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:18.855733 22805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:18.855865 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22738
I20260621 07:09:18.856051 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:18.856618 1779 external_mini_cluster.cc:1428] Setting key f593726dad461621d302f77f62f4b864
I20260621 07:09:18.864252 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:18.858428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55955 (local address 127.0.0.1:36097)
0621 07:09:18.859323 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:18.859334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:18.859372 (+ 38us) server_negotiation.cc:407] Connection header received
0621 07:09:18.860894 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:18.860914 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:18.861243 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:18.861563 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:18.862619 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:18.862625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:18.862637 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:18.862650 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:18.862710 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:18.862731 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:18.862893 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:18.862994 (+ 101us) server_negotiation.cc:299] Negotiation successful
0621 07:09:18.863197 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":410,"threads_started":1}
I20260621 07:09:18.867785 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:18.869361 22805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0: Bootstrap starting.
I20260621 07:09:18.873459 22805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:18.874883 22805 log.cc:824] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:18.875907 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:18.873262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39147 (local address 127.0.0.1:36097)
0621 07:09:18.873477 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:18.873485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:18.873854 (+ 369us) server_negotiation.cc:407] Connection header received
0621 07:09:18.874003 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:18.874009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:18.874070 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:18.874180 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:18.875141 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:18.875146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:18.875151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:18.875169 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:18.875217 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:18.875235 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:18.875361 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:18.875515 (+ 154us) server_negotiation.cc:299] Negotiation successful
0621 07:09:18.875587 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:09:18.878217 22805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0: No bootstrap required, opened a new log
I20260621 07:09:18.885905 22805 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } }
I20260621 07:09:18.886307 22805 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:18.886426 22805 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e95be71fbe4a0fa4fe483bbd9853e0, State: Initialized, Role: FOLLOWER
I20260621 07:09:18.887038 22805 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } }
I20260621 07:09:18.887276 22805 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:18.887459 22805 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:18.887626 22805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:18.890225 22805 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } }
I20260621 07:09:18.890727 22805 leader_election.cc:302] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e95be71fbe4a0fa4fe483bbd9853e0; no voters:
I20260621 07:09:18.891358 22805 leader_election.cc:288] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:18.891561 22815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:18.892247 22815 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [term 1 LEADER]: Becoming Leader. State: Replica: a8e95be71fbe4a0fa4fe483bbd9853e0, State: Running, Role: LEADER
I20260621 07:09:18.892796 22815 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } }
I20260621 07:09:18.893497 22805 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:18.896721 22816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } } }
I20260621 07:09:18.897202 22816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:18.897050 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e95be71fbe4a0fa4fe483bbd9853e0. Latest consensus state: current_term: 1 leader_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e95be71fbe4a0fa4fe483bbd9853e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36097 } } }
I20260621 07:09:18.898623 22823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:18.898630 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:18.904242 22823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:18.914014 22823 catalog_manager.cc:1347] Generated new cluster ID: 5a9e198c12da424fba5a2cfb6cc4cb43
I20260621 07:09:18.914136 22823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:18.936257 22823 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:18.936690 22823 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:18.961052 22823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8e95be71fbe4a0fa4fe483bbd9853e0: Generated new TSK 0
I20260621 07:09:18.961930 22823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:18.974298 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22738
W20260621 07:09:18.975508 22800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55955
W20260621 07:09:18.976190 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:18.976346 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22738
2026-06-21T07:09:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:19 dist-test-slave-hlq1 krb5kdc[22836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:19 dist-test-slave-hlq1 krb5kdc[22836](info): set up 2 sockets
Jun 21 07:09:19 dist-test-slave-hlq1 krb5kdc[22836](info): commencing operation
krb5kdc: starting...
W20260621 07:09:20.108362 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:20 dist-test-slave-hlq1 krb5kdc[22836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:20.417227 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:20.802438 22849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:20.802810 22849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:20.803040 22849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:20.810914 22849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:20.811055 22849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:20.811172 22849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:20.811306 22849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:20.811390 22849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:20.822588 22849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:20.824368 22849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:20.826002 22849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:20.840009 22857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:20.840031 22854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:20.839958 22855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:20.841341 22849 server_base.cc:1034] running on GCE node
I20260621 07:09:20.842402 22849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:20.844157 22849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:20.845376 22849 hybrid_clock.cc:648] HybridClock initialized: now 1782025760845314 us; error 42 us; skew 500 ppm
I20260621 07:09:20.845781 22849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:20 dist-test-slave-hlq1 krb5kdc[22836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:20.849428 22849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:20.851145 22849 webserver.cc:460] Webserver started at http://127.0.0.1:33389/ using document root <none> and password file <none>
I20260621 07:09:20.852041 22849 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:20.852151 22849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:20.852459 22849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:20.856124 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "657c8929dcc14477ba9e77eccdf29f21"
format_stamp: "Formatted at 2026-06-21 07:09:20 on dist-test-slave-hlq1"
server_key: "33f6a12fdfee5943d33782c15899a676"
server_key_iv: "b9763a30fa15d9c3671d2fdbd9a05e1c"
server_key_version: "encryptionkey@0"
I20260621 07:09:20.856982 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "657c8929dcc14477ba9e77eccdf29f21"
format_stamp: "Formatted at 2026-06-21 07:09:20 on dist-test-slave-hlq1"
server_key: "33f6a12fdfee5943d33782c15899a676"
server_key_iv: "b9763a30fa15d9c3671d2fdbd9a05e1c"
server_key_version: "encryptionkey@0"
I20260621 07:09:20.863829 22849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260621 07:09:20.868460 22864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:20.871134 22849 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260621 07:09:20.871590 22849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "657c8929dcc14477ba9e77eccdf29f21"
format_stamp: "Formatted at 2026-06-21 07:09:20 on dist-test-slave-hlq1"
server_key: "33f6a12fdfee5943d33782c15899a676"
server_key_iv: "b9763a30fa15d9c3671d2fdbd9a05e1c"
server_key_version: "encryptionkey@0"
I20260621 07:09:20.871917 22849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:20.905584 22849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:20.910657 22849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:20.911003 22849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:20.932405 22849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38253
I20260621 07:09:20.932431 22915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38253 every 8 connection(s)
I20260621 07:09:20.934213 22849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:20.939378 22916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:20.940364 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22849
I20260621 07:09:20.940610 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:20.941140 1779 external_mini_cluster.cc:1428] Setting key 19dc8b05f5c47369f91da8eb72b38c5c
I20260621 07:09:20.955495 22916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21: Bootstrap starting.
Jun 21 07:09:20 dist-test-slave-hlq1 krb5kdc[22836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:09:20.960479 22916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:20.962273 22916 log.cc:824] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:20.966377 22916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21: No bootstrap required, opened a new log
I20260621 07:09:20.969210 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:20.943324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:33677 (local address 127.0.0.1:38253)
0621 07:09:20.944283 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:20.944297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:20.944346 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:09:20.946426 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:20.946455 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:20.946980 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:20.947379 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:20.949493 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:20.950936 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:20.952908 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:20.953701 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:20.953858 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:20.959414 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:20.959429 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:20.959452 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:20.959519 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:20.963772 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:20.964734 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:20.964744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:20.964753 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:20.964995 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:20.965502 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:20.965506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:20.965509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:20.966301 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:20.966693 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:20.967210 (+ 517us) server_negotiation.cc:299] Negotiation successful
0621 07:09:20.967648 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":401,"threads_started":1}
I20260621 07:09:20.973958 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:20.976862 22916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } }
I20260621 07:09:20.977528 22916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:20.977665 22916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657c8929dcc14477ba9e77eccdf29f21, State: Initialized, Role: FOLLOWER
I20260621 07:09:20.978423 22916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } }
I20260621 07:09:20.978711 22916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:20.978873 22916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:20.979120 22916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:20.982837 22916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } }
I20260621 07:09:20.983568 22916 leader_election.cc:302] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 657c8929dcc14477ba9e77eccdf29f21; no voters:
I20260621 07:09:20.984323 22916 leader_election.cc:288] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:20.984432 22926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:20.986052 22926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [term 1 LEADER]: Becoming Leader. State: Replica: 657c8929dcc14477ba9e77eccdf29f21, State: Running, Role: LEADER
I20260621 07:09:20.986841 22926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } }
I20260621 07:09:20.987023 22916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:20.990433 22927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "657c8929dcc14477ba9e77eccdf29f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } } }
I20260621 07:09:20.990731 22927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:20.989992 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657c8929dcc14477ba9e77eccdf29f21. Latest consensus state: current_term: 1 leader_uuid: "657c8929dcc14477ba9e77eccdf29f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657c8929dcc14477ba9e77eccdf29f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38253 } } }
I20260621 07:09:20.991719 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:21.000629 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:20.979091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:34565 (local address 127.0.0.1:38253)
0621 07:09:20.979340 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:20.979350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:20.979905 (+ 555us) server_negotiation.cc:407] Connection header received
0621 07:09:20.980094 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:20.980099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:20.980179 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:20.980274 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:20.982271 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:20.983105 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:20.985118 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:20.985764 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:20.985873 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:20.990176 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:20.990187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:20.990196 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:20.990261 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:20.995301 (+ 5040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:20.996758 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:20.996768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:20.996774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:20.996971 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:20.997720 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:20.997729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:20.997735 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:20.998336 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:20.998740 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:20.999755 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0621 07:09:21.000046 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 07:09:21.003569 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:21.010234 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:21.021412 22933 catalog_manager.cc:1347] Generated new cluster ID: 2786db7d35c3402fbc7382d3f7fa4e16
I20260621 07:09:21.021564 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:21.039690 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:21.041173 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:21.055884 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 657c8929dcc14477ba9e77eccdf29f21: Generated new TSK 0
I20260621 07:09:21.057001 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:21.064404 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22849
I20260621 07:09:21.065901 22911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:21.168300 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22849
W20260621 07:09:21.188957 22834 connection.cc:535] client connection to 127.0.0.1:38253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38253): Connection reset by peer (error 104)
2026-06-21T07:09:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:21 dist-test-slave-hlq1 krb5kdc[22948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:21 dist-test-slave-hlq1 krb5kdc[22948](info): set up 2 sockets
Jun 21 07:09:21 dist-test-slave-hlq1 krb5kdc[22948](info): commencing operation
krb5kdc: starting...
W20260621 07:09:22.304354 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:22 dist-test-slave-hlq1 krb5kdc[22948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:22.564703 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:22.918828 22961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:22.919148 22961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:22.919312 22961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:22.926890 22961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:22.927022 22961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:22.927112 22961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:22.927228 22961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:22.927307 22961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:22.937907 22961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:22.939323 22961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:22.940687 22961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:22.952603 22969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:22.952725 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:22.952661 22967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:22.953871 22961 server_base.cc:1034] running on GCE node
I20260621 07:09:22.954768 22961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:22.956266 22961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:22.957528 22961 hybrid_clock.cc:648] HybridClock initialized: now 1782025762957475 us; error 57 us; skew 500 ppm
I20260621 07:09:22.957937 22961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:22 dist-test-slave-hlq1 krb5kdc[22948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:22.964318 22961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:22.965961 22961 webserver.cc:460] Webserver started at http://127.0.0.1:41695/ using document root <none> and password file <none>
I20260621 07:09:22.966699 22961 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:22.966853 22961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:22.967177 22961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:22.970395 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "fe2d9e57703041f8bd73725e6c20d4a9"
format_stamp: "Formatted at 2026-06-21 07:09:22 on dist-test-slave-hlq1"
server_key: "3bb1e429eb79f12bf0f47aaba1a4e946"
server_key_iv: "0373018c7ca24c3307a783e3fa3e4096"
server_key_version: "encryptionkey@0"
I20260621 07:09:22.971197 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "fe2d9e57703041f8bd73725e6c20d4a9"
format_stamp: "Formatted at 2026-06-21 07:09:22 on dist-test-slave-hlq1"
server_key: "3bb1e429eb79f12bf0f47aaba1a4e946"
server_key_iv: "0373018c7ca24c3307a783e3fa3e4096"
server_key_version: "encryptionkey@0"
I20260621 07:09:22.976588 22961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260621 07:09:22.980118 22976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:22.981488 22961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:22.981679 22961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "fe2d9e57703041f8bd73725e6c20d4a9"
format_stamp: "Formatted at 2026-06-21 07:09:22 on dist-test-slave-hlq1"
server_key: "3bb1e429eb79f12bf0f47aaba1a4e946"
server_key_iv: "0373018c7ca24c3307a783e3fa3e4096"
server_key_version: "encryptionkey@0"
I20260621 07:09:22.981894 22961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:23.002233 22961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:23.006541 22961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:23.006855 22961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:23.025743 22961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40215
I20260621 07:09:23.025799 23027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40215 every 8 connection(s)
I20260621 07:09:23.027171 22961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:23.030267 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 22961
I20260621 07:09:23.030500 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:23.031006 1779 external_mini_cluster.cc:1428] Setting key 119bce03c153db01dade50818b8ec36c
I20260621 07:09:23.031487 23028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:23.046391 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:23.032853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:41459 (local address 127.0.0.1:40215)
0621 07:09:23.033656 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:23.033671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:23.033720 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:09:23.036284 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:23.036313 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:23.036822 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:23.037248 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:23.037759 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:23.039812 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:23.041197 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:23.042250 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:23.042438 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:23.043654 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:23.043663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:23.043680 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:23.044422 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:23.044514 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:23.044544 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:23.044693 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:23.044948 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 07:09:23.045116 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":336,"threads_started":1}
I20260621 07:09:23.046594 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9: Bootstrap starting.
I20260621 07:09:23.050083 23028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:23.050161 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:23.051112 23028 log.cc:824] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:23.053997 23028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9: No bootstrap required, opened a new log
I20260621 07:09:23.062006 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:23.055117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:60859 (local address 127.0.0.1:40215)
0621 07:09:23.055433 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:23.055440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:23.055788 (+ 348us) server_negotiation.cc:407] Connection header received
0621 07:09:23.055963 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:23.055966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:23.056027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:23.056110 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:23.056647 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:23.057444 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:23.059294 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:23.059859 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:23.059954 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:23.061180 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:23.061184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:23.061188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:23.061216 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:23.061263 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:23.061279 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:23.061368 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:23.061570 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 07:09:23.061648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260621 07:09:23.063858 23028 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260621 07:09:23.064241 23028 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:23.064350 23028 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2d9e57703041f8bd73725e6c20d4a9, State: Initialized, Role: FOLLOWER
I20260621 07:09:23.064970 23028 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260621 07:09:23.065251 23028 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:23.065399 23028 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:23.065580 23028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:23.067833 23028 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260621 07:09:23.068325 23028 leader_election.cc:302] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2d9e57703041f8bd73725e6c20d4a9; no voters:
I20260621 07:09:23.068929 23028 leader_election.cc:288] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:23.068994 23038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:23.071008 23028 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:23.071738 23038 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [term 1 LEADER]: Becoming Leader. State: Replica: fe2d9e57703041f8bd73725e6c20d4a9, State: Running, Role: LEADER
I20260621 07:09:23.072243 23038 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } }
I20260621 07:09:23.075485 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2d9e57703041f8bd73725e6c20d4a9. Latest consensus state: current_term: 1 leader_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } } }
I20260621 07:09:23.075420 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2d9e57703041f8bd73725e6c20d4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40215 } } }
I20260621 07:09:23.075872 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:23.075942 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:23.079083 23050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:23.083235 23050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:23.091564 23050 catalog_manager.cc:1347] Generated new cluster ID: 9f00b90c1f34458ca529651957b3496a
I20260621 07:09:23.091673 23050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:23.104368 23050 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:23.106185 23050 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:23.118412 23050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2d9e57703041f8bd73725e6c20d4a9: Generated new TSK 0
I20260621 07:09:23.119319 23050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:23.127678 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22961
W20260621 07:09:23.129048 23023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:41459
W20260621 07:09:23.129866 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 22961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:23.130019 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 22961
W20260621 07:09:23.147135 22946 connection.cc:535] client connection to 127.0.0.1:40215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40215): Connection reset by peer (error 104)
2026-06-21T07:09:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:23 dist-test-slave-hlq1 krb5kdc[23059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:23 dist-test-slave-hlq1 krb5kdc[23059](info): set up 2 sockets
Jun 21 07:09:23 dist-test-slave-hlq1 krb5kdc[23059](info): commencing operation
krb5kdc: starting...
W20260621 07:09:24.240653 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:24 dist-test-slave-hlq1 krb5kdc[23059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:24.547822 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:24.938730 23072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:24.939167 23072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:24.939368 23072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:24.947280 23072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:24.947510 23072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:24.947645 23072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:24.947763 23072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:24.947835 23072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:24.958796 23072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:24.960407 23072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:24.961968 23072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:24.975786 23077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:24.975746 23078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:24.975683 23080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:24.977150 23072 server_base.cc:1034] running on GCE node
I20260621 07:09:24.978041 23072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:24.979768 23072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:24.981021 23072 hybrid_clock.cc:648] HybridClock initialized: now 1782025764980965 us; error 61 us; skew 500 ppm
I20260621 07:09:24.981472 23072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:24 dist-test-slave-hlq1 krb5kdc[23059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:24.985319 23072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:24.987195 23072 webserver.cc:460] Webserver started at http://127.0.0.1:32935/ using document root <none> and password file <none>
I20260621 07:09:24.988157 23072 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:24.988327 23072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:24.988706 23072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:24.992494 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "7eb2047c867a408ca8521b55f3eb06c2"
format_stamp: "Formatted at 2026-06-21 07:09:24 on dist-test-slave-hlq1"
server_key: "8539e8c6c69e4bf37ab5ae5e870c94a1"
server_key_iv: "0d872d657bfc1ee663b6c2edb5337578"
server_key_version: "encryptionkey@0"
I20260621 07:09:24.993407 23072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "7eb2047c867a408ca8521b55f3eb06c2"
format_stamp: "Formatted at 2026-06-21 07:09:24 on dist-test-slave-hlq1"
server_key: "8539e8c6c69e4bf37ab5ae5e870c94a1"
server_key_iv: "0d872d657bfc1ee663b6c2edb5337578"
server_key_version: "encryptionkey@0"
I20260621 07:09:24.999650 23072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 07:09:25.003681 23087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:25.005240 23072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:25.005465 23072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "7eb2047c867a408ca8521b55f3eb06c2"
format_stamp: "Formatted at 2026-06-21 07:09:24 on dist-test-slave-hlq1"
server_key: "8539e8c6c69e4bf37ab5ae5e870c94a1"
server_key_iv: "0d872d657bfc1ee663b6c2edb5337578"
server_key_version: "encryptionkey@0"
I20260621 07:09:25.005702 23072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:25.012701 23072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:25.018038 23072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:25.018402 23072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:25.040225 23072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35169
I20260621 07:09:25.040288 23138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35169 every 8 connection(s)
I20260621 07:09:25.041924 23072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:25.046623 23139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:25.050164 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23072
I20260621 07:09:25.050393 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:25.051047 1779 external_mini_cluster.cc:1428] Setting key af13c2ececb461d9509f8474ad26be8b
I20260621 07:09:25.061379 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:25.053403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:56301 (local address 127.0.0.1:35169)
0621 07:09:25.054450 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:25.054465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:25.054514 (+ 49us) server_negotiation.cc:407] Connection header received
0621 07:09:25.057079 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:25.057109 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:25.057533 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:25.058071 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:25.059370 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:25.059378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:25.059447 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:25.059477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:25.059568 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:25.059596 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:25.059880 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:25.060052 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 07:09:25.060277 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":450,"threads_started":1}
I20260621 07:09:25.061841 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2: Bootstrap starting.
I20260621 07:09:25.065843 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:25.066126 23139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:25.067284 23139 log.cc:824] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:25.070995 23139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2: No bootstrap required, opened a new log
I20260621 07:09:25.074152 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:25.071146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:51661 (local address 127.0.0.1:35169)
0621 07:09:25.071351 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:25.071356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:25.071933 (+ 577us) server_negotiation.cc:407] Connection header received
0621 07:09:25.072091 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:25.072099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:25.072170 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:25.072290 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:25.073418 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:25.073426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:25.073432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:25.073458 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:25.073520 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:25.073542 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:25.073682 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:25.073741 (+ 59us) server_negotiation.cc:299] Negotiation successful
0621 07:09:25.073813 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:09:25.080706 23139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } }
I20260621 07:09:25.081189 23139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:25.081300 23139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb2047c867a408ca8521b55f3eb06c2, State: Initialized, Role: FOLLOWER
I20260621 07:09:25.082046 23139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } }
I20260621 07:09:25.082314 23139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:25.082485 23139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:25.082690 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:25.085750 23139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } }
I20260621 07:09:25.086352 23139 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb2047c867a408ca8521b55f3eb06c2; no voters:
I20260621 07:09:25.086961 23139 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:25.087158 23149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:25.087910 23149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb2047c867a408ca8521b55f3eb06c2, State: Running, Role: LEADER
I20260621 07:09:25.088604 23149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } }
I20260621 07:09:25.089685 23139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:25.092263 23151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb2047c867a408ca8521b55f3eb06c2. Latest consensus state: current_term: 1 leader_uuid: "7eb2047c867a408ca8521b55f3eb06c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } } }
I20260621 07:09:25.092792 23151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:25.092391 23150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb2047c867a408ca8521b55f3eb06c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb2047c867a408ca8521b55f3eb06c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35169 } } }
I20260621 07:09:25.093489 23150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:25.094830 23156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:25.102353 23156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:25.113345 23156 catalog_manager.cc:1347] Generated new cluster ID: a89e6504a2954a52886344fd4e5ba4df
I20260621 07:09:25.113494 23156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:25.145614 23156 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:25.146220 23156 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:25.156306 23156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb2047c867a408ca8521b55f3eb06c2: Generated new TSK 0
I20260621 07:09:25.157286 23156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:25.171057 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23072
W20260621 07:09:25.172253 23134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:56301
W20260621 07:09:25.172994 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23072: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:25.173161 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23072
2026-06-21T07:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:25 dist-test-slave-hlq1 krb5kdc[23170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:25 dist-test-slave-hlq1 krb5kdc[23170](info): set up 2 sockets
Jun 21 07:09:25 dist-test-slave-hlq1 krb5kdc[23170](info): commencing operation
krb5kdc: starting...
W20260621 07:09:26.288354 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:26 dist-test-slave-hlq1 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:26.583374 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:26.969295 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:26.969709 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:26.969882 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:26.977785 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:26.977977 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:26.978089 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:26.978211 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:26.978295 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:26.989003 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:26.990664 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:26.992321 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:27.005681 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:27.005849 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:27.005739 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:27.007093 23183 server_base.cc:1034] running on GCE node
I20260621 07:09:27.008034 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:27.009838 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:27.011098 23183 hybrid_clock.cc:648] HybridClock initialized: now 1782025767011033 us; error 50 us; skew 500 ppm
I20260621 07:09:27.011591 23183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:27 dist-test-slave-hlq1 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:27.015105 23183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:27.017057 23183 webserver.cc:460] Webserver started at http://127.0.0.1:45129/ using document root <none> and password file <none>
I20260621 07:09:27.017907 23183 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:27.018061 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:27.018419 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:27.021934 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "ed9d78287df24620802f66e5d8531f7c"
format_stamp: "Formatted at 2026-06-21 07:09:27 on dist-test-slave-hlq1"
server_key: "430c974e9a60c0cf589183e6d15183f7"
server_key_iv: "7b04e97326b2bfe412f8c70f8f4c1ed3"
server_key_version: "encryptionkey@0"
I20260621 07:09:27.022804 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "ed9d78287df24620802f66e5d8531f7c"
format_stamp: "Formatted at 2026-06-21 07:09:27 on dist-test-slave-hlq1"
server_key: "430c974e9a60c0cf589183e6d15183f7"
server_key_iv: "7b04e97326b2bfe412f8c70f8f4c1ed3"
server_key_version: "encryptionkey@0"
I20260621 07:09:27.029126 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260621 07:09:27.033490 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:27.035463 23183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260621 07:09:27.035712 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "ed9d78287df24620802f66e5d8531f7c"
format_stamp: "Formatted at 2026-06-21 07:09:27 on dist-test-slave-hlq1"
server_key: "430c974e9a60c0cf589183e6d15183f7"
server_key_iv: "7b04e97326b2bfe412f8c70f8f4c1ed3"
server_key_version: "encryptionkey@0"
I20260621 07:09:27.035986 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:27.055558 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:27.060395 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:27.060727 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:27.082417 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38905
I20260621 07:09:27.082442 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38905 every 8 connection(s)
I20260621 07:09:27.084208 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:27.088858 23250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:27.093729 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23183
I20260621 07:09:27.093946 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:27.094462 1779 external_mini_cluster.cc:1428] Setting key 6926bd64b04aeae572bba9ccfb7ba9dd
I20260621 07:09:27.102587 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c: Bootstrap starting.
I20260621 07:09:27.106845 23250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:27.108392 23250 log.cc:824] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:27.111868 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c: No bootstrap required, opened a new log
Jun 21 07:09:27 dist-test-slave-hlq1 krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 07:09:27.121639 23250 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260621 07:09:27.122077 23250 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:27.122112 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:27.096469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:48045 (local address 127.0.0.1:38905)
0621 07:09:27.097338 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:27.097357 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:27.097414 (+ 57us) server_negotiation.cc:407] Connection header received
0621 07:09:27.100589 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:27.100623 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:27.101209 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:27.101675 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:27.103149 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:27.105315 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:27.106715 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:27.107881 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:27.108086 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:27.113288 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:27.113297 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:27.113312 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:27.113361 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:27.116947 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:27.118007 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:27.118015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:27.118021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:27.118213 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:27.118763 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:27.118767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:27.118769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:27.119543 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:27.119840 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:27.120397 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 07:09:27.120809 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":350,"threads_started":1}
I20260621 07:09:27.122213 23250 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9d78287df24620802f66e5d8531f7c, State: Initialized, Role: FOLLOWER
I20260621 07:09:27.122948 23250 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260621 07:09:27.123230 23250 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:27.123390 23250 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:27.123697 23250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:27.126509 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:27.126621 23250 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260621 07:09:27.127229 23250 leader_election.cc:302] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed9d78287df24620802f66e5d8531f7c; no voters:
I20260621 07:09:27.127991 23250 leader_election.cc:288] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:27.128119 23255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:27.130849 23250 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:27.135160 23255 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [term 1 LEADER]: Becoming Leader. State: Replica: ed9d78287df24620802f66e5d8531f7c, State: Running, Role: LEADER
I20260621 07:09:27.136070 23255 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260621 07:09:27.146319 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed9d78287df24620802f66e5d8531f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } } }
I20260621 07:09:27.146842 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:27.150890 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:27.132049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:51547 (local address 127.0.0.1:38905)
0621 07:09:27.132239 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:27.132244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:27.133087 (+ 843us) server_negotiation.cc:407] Connection header received
0621 07:09:27.133312 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:27.133318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:27.133400 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:27.133511 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:27.135046 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:27.136022 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:27.137490 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:27.137985 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:27.138129 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:27.141488 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:27.141499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:27.141508 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:27.141559 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:27.145511 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:27.146612 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:27.146617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:27.146620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:27.146783 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:27.147429 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:27.147434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:27.147437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:27.147936 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:27.149823 (+ 1887us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:27.150065 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 07:09:27.150340 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:09:27.151278 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed9d78287df24620802f66e5d8531f7c. Latest consensus state: current_term: 1 leader_uuid: "ed9d78287df24620802f66e5d8531f7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed9d78287df24620802f66e5d8531f7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } } }
I20260621 07:09:27.152364 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:27.156024 23269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:27.161700 23269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:27.172338 23269 catalog_manager.cc:1347] Generated new cluster ID: c5fe2687072c4ed6a2940a6cc7b7c458
I20260621 07:09:27.172482 23269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:27.183477 23269 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:27.184993 23269 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:27.197350 23269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed9d78287df24620802f66e5d8531f7c: Generated new TSK 0
I20260621 07:09:27.198859 23269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:27.212347 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23183
I20260621 07:09:27.214206 23245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:27.312482 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23183
W20260621 07:09:27.332175 23168 connection.cc:535] client connection to 127.0.0.1:38905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38905): Connection reset by peer (error 104)
2026-06-21T07:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:27 dist-test-slave-hlq1 krb5kdc[23282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:27 dist-test-slave-hlq1 krb5kdc[23282](info): set up 2 sockets
Jun 21 07:09:27 dist-test-slave-hlq1 krb5kdc[23282](info): commencing operation
krb5kdc: starting...
W20260621 07:09:28.456398 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:28 dist-test-slave-hlq1 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:28.755266 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:29.142169 23295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:29.142546 23295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:29.142733 23295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:29.150718 23295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:29.150876 23295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:29.150965 23295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:29.151083 23295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:29.151167 23295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:29.162168 23295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:29.164085 23295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:29.165755 23295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:29.179137 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:29.179137 23303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:29.180269 23295 server_base.cc:1034] running on GCE node
W20260621 07:09:29.181318 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:29.182492 23295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:29.184478 23295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:29.185732 23295 hybrid_clock.cc:648] HybridClock initialized: now 1782025769185687 us; error 36 us; skew 500 ppm
I20260621 07:09:29.186151 23295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:29 dist-test-slave-hlq1 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:29.189847 23295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:29.191598 23295 webserver.cc:460] Webserver started at http://127.0.0.1:36823/ using document root <none> and password file <none>
I20260621 07:09:29.192395 23295 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:29.192561 23295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:29.192924 23295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:29.197299 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "494b0b2ee6e541e09d3e9724889ea908"
format_stamp: "Formatted at 2026-06-21 07:09:29 on dist-test-slave-hlq1"
server_key: "7d6da4be81210f7cc5e37657ee111204"
server_key_iv: "c5b336519863ffe67f32800b66296143"
server_key_version: "encryptionkey@0"
I20260621 07:09:29.198549 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "494b0b2ee6e541e09d3e9724889ea908"
format_stamp: "Formatted at 2026-06-21 07:09:29 on dist-test-slave-hlq1"
server_key: "7d6da4be81210f7cc5e37657ee111204"
server_key_iv: "c5b336519863ffe67f32800b66296143"
server_key_version: "encryptionkey@0"
I20260621 07:09:29.206527 23295 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20260621 07:09:29.210716 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:29.212291 23295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:09:29.212499 23295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "494b0b2ee6e541e09d3e9724889ea908"
format_stamp: "Formatted at 2026-06-21 07:09:29 on dist-test-slave-hlq1"
server_key: "7d6da4be81210f7cc5e37657ee111204"
server_key_iv: "c5b336519863ffe67f32800b66296143"
server_key_version: "encryptionkey@0"
I20260621 07:09:29.212736 23295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:29.245275 23295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:29.250159 23295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:29.250468 23295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:29.272203 23295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43475
I20260621 07:09:29.272236 23361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43475 every 8 connection(s)
I20260621 07:09:29.273878 23295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:29.276253 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23295
I20260621 07:09:29.276458 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:29.277110 1779 external_mini_cluster.cc:1428] Setting key 57478e94ab0b2556efc95c7dc43b382e
I20260621 07:09:29.279241 23362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:29.292096 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:29.279362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:34201 (local address 127.0.0.1:43475)
0621 07:09:29.280216 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:29.280230 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:29.280276 (+ 46us) server_negotiation.cc:407] Connection header received
0621 07:09:29.282307 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:29.282333 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:29.282819 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:29.283290 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:29.283929 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:29.285407 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:29.286845 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:29.287641 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:29.287845 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:29.289339 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:29.289348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:29.289369 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:29.290053 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:29.290152 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:29.290186 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:29.290369 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:29.290666 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 07:09:29.290859 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":342,"threads_started":1}
I20260621 07:09:29.292748 23362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908: Bootstrap starting.
I20260621 07:09:29.296691 23362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:29.296752 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:29.297869 23362 log.cc:824] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:29.301246 23362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908: No bootstrap required, opened a new log
I20260621 07:09:29.309123 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:29.301551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:47765 (local address 127.0.0.1:43475)
0621 07:09:29.301772 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:29.301781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:29.302366 (+ 585us) server_negotiation.cc:407] Connection header received
0621 07:09:29.302593 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:29.302597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:29.302668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:29.302747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:29.303483 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:29.304358 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:29.306314 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:29.306760 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:29.306860 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:29.308267 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:29.308272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:29.308275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:29.308294 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:29.308339 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:29.308358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:29.308445 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:29.308652 (+ 207us) server_negotiation.cc:299] Negotiation successful
0621 07:09:29.308704 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:09:29.312083 23362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260621 07:09:29.312546 23362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:29.312662 23362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494b0b2ee6e541e09d3e9724889ea908, State: Initialized, Role: FOLLOWER
I20260621 07:09:29.313316 23362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260621 07:09:29.313596 23362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:29.313747 23362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:29.313948 23362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:29.316708 23362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260621 07:09:29.317358 23362 leader_election.cc:302] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 494b0b2ee6e541e09d3e9724889ea908; no voters:
I20260621 07:09:29.317974 23362 leader_election.cc:288] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:29.318125 23372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:29.318940 23372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [term 1 LEADER]: Becoming Leader. State: Replica: 494b0b2ee6e541e09d3e9724889ea908, State: Running, Role: LEADER
I20260621 07:09:29.319592 23372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260621 07:09:29.320423 23362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:29.323078 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "494b0b2ee6e541e09d3e9724889ea908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260621 07:09:29.323187 23374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494b0b2ee6e541e09d3e9724889ea908. Latest consensus state: current_term: 1 leader_uuid: "494b0b2ee6e541e09d3e9724889ea908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b0b2ee6e541e09d3e9724889ea908" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260621 07:09:29.323565 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:29.323589 23374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:29.325683 23377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:29.331943 23377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:29.343632 23377 catalog_manager.cc:1347] Generated new cluster ID: 07c6604bb7ca4f3e90fb5c85fe3f64b0
I20260621 07:09:29.343787 23377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:29.382449 23377 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:29.384392 23377 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:29.392181 23377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494b0b2ee6e541e09d3e9724889ea908: Generated new TSK 0
I20260621 07:09:29.393201 23377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:29.410758 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23295
W20260621 07:09:29.413187 23357 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:34201
W20260621 07:09:29.413985 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:29.414152 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23295
W20260621 07:09:29.432025 23280 connection.cc:535] client connection to 127.0.0.1:43475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43475): Connection reset by peer (error 104)
2026-06-21T07:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2099 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:29 dist-test-slave-hlq1 krb5kdc[23393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:29 dist-test-slave-hlq1 krb5kdc[23393](info): set up 2 sockets
Jun 21 07:09:29 dist-test-slave-hlq1 krb5kdc[23393](info): commencing operation
krb5kdc: starting...
W20260621 07:09:30.528340 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:30 dist-test-slave-hlq1 krb5kdc[23393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 07:09:30.822553 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:31.203858 23406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:31.204192 23406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:31.204376 23406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:31.212126 23406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:31.212255 23406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:31.212357 23406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:31.212467 23406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:31.212553 23406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:31.224056 23406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:31.225787 23406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:31.227391 23406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:31.240990 23414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:31.241011 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:31.241173 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:31.244987 23406 server_base.cc:1034] running on GCE node
I20260621 07:09:31.245913 23406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:31.247671 23406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:31.248885 23406 hybrid_clock.cc:648] HybridClock initialized: now 1782025771248847 us; error 33 us; skew 500 ppm
I20260621 07:09:31.249251 23406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:31 dist-test-slave-hlq1 krb5kdc[23393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:31.252964 23406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 07:09:31.255028 23406 webserver.cc:460] Webserver started at http://127.0.0.1:36331/ using document root <none> and password file <none>
I20260621 07:09:31.255929 23406 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:31.256071 23406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:31.256438 23406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:31.260043 23406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "cffaf92a17844473879e5bc92397b6c8"
format_stamp: "Formatted at 2026-06-21 07:09:31 on dist-test-slave-hlq1"
server_key: "c55c55e178949f5f1f039def9741e3e3"
server_key_iv: "6196ee874374f8344fe2b90fc6525c42"
server_key_version: "encryptionkey@0"
I20260621 07:09:31.260895 23406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "cffaf92a17844473879e5bc92397b6c8"
format_stamp: "Formatted at 2026-06-21 07:09:31 on dist-test-slave-hlq1"
server_key: "c55c55e178949f5f1f039def9741e3e3"
server_key_iv: "6196ee874374f8344fe2b90fc6525c42"
server_key_version: "encryptionkey@0"
I20260621 07:09:31.266778 23406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:09:31.270697 23421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:31.272231 23406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:09:31.272423 23406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "cffaf92a17844473879e5bc92397b6c8"
format_stamp: "Formatted at 2026-06-21 07:09:31 on dist-test-slave-hlq1"
server_key: "c55c55e178949f5f1f039def9741e3e3"
server_key_iv: "6196ee874374f8344fe2b90fc6525c42"
server_key_version: "encryptionkey@0"
I20260621 07:09:31.272652 23406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:31.279244 23406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:31.284305 23406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:31.284639 23406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:31.305116 23406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35599
I20260621 07:09:31.305178 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35599 every 8 connection(s)
I20260621 07:09:31.306739 23406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:31.311172 23473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:31.312536 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23406
I20260621 07:09:31.312743 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:31.313256 1779 external_mini_cluster.cc:1428] Setting key ef767fcb52beb5753529b7c5bd6bc9c9
I20260621 07:09:31.322124 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:31.315647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:57441 (local address 127.0.0.1:35599)
0621 07:09:31.316626 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:31.316639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:31.316693 (+ 54us) server_negotiation.cc:407] Connection header received
0621 07:09:31.318432 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:31.318458 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:31.318859 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:31.319242 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:31.320428 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:31.320435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:31.320448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:31.320461 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:31.320529 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:31.320552 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:31.320726 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:31.320831 (+ 105us) server_negotiation.cc:299] Negotiation successful
0621 07:09:31.321073 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":467,"threads_started":1}
I20260621 07:09:31.326046 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:31.326273 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8: Bootstrap starting.
I20260621 07:09:31.329806 23473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:31.331180 23473 log.cc:824] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:31.333878 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:31.331030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:52723 (local address 127.0.0.1:35599)
0621 07:09:31.331267 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:31.331276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:31.331655 (+ 379us) server_negotiation.cc:407] Connection header received
0621 07:09:31.331867 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:31.331874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:31.331951 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:31.332094 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:31.333140 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:31.333145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:31.333149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:31.333164 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:31.333207 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:31.333230 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:31.333340 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:31.333478 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 07:09:31.333539 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260621 07:09:31.334872 23473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8: No bootstrap required, opened a new log
I20260621 07:09:31.342774 23473 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260621 07:09:31.343252 23473 raft_consensus.cc:374] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:31.343369 23473 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffaf92a17844473879e5bc92397b6c8, State: Initialized, Role: FOLLOWER
I20260621 07:09:31.344105 23473 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260621 07:09:31.344370 23473 raft_consensus.cc:388] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:31.344537 23473 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:31.344741 23473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:31.347148 23473 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260621 07:09:31.347808 23473 leader_election.cc:302] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cffaf92a17844473879e5bc92397b6c8; no voters:
I20260621 07:09:31.348536 23473 leader_election.cc:288] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:31.348713 23483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:31.349795 23483 raft_consensus.cc:686] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [term 1 LEADER]: Becoming Leader. State: Replica: cffaf92a17844473879e5bc92397b6c8, State: Running, Role: LEADER
I20260621 07:09:31.350396 23483 consensus_queue.cc:237] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } }
I20260621 07:09:31.351040 23473 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:31.354496 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cffaf92a17844473879e5bc92397b6c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } } }
I20260621 07:09:31.355011 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:31.354576 23485 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cffaf92a17844473879e5bc92397b6c8. Latest consensus state: current_term: 1 leader_uuid: "cffaf92a17844473879e5bc92397b6c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffaf92a17844473879e5bc92397b6c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35599 } } }
I20260621 07:09:31.355219 23485 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:31.360592 23491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:31.366884 23491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:31.378458 23491 catalog_manager.cc:1347] Generated new cluster ID: df2ae3513d20494bac91f4e9a84b79b1
I20260621 07:09:31.378644 23491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:31.396926 23491 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:31.397490 23491 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:31.405718 23491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cffaf92a17844473879e5bc92397b6c8: Generated new TSK 0
I20260621 07:09:31.406595 23491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:31.412226 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23406
W20260621 07:09:31.413314 23468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:57441
W20260621 07:09:31.413998 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:31.414149 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23406
2026-06-21T07:09:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:31 dist-test-slave-hlq1 krb5kdc[23504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:31 dist-test-slave-hlq1 krb5kdc[23504](info): set up 2 sockets
Jun 21 07:09:31 dist-test-slave-hlq1 krb5kdc[23504](info): commencing operation
krb5kdc: starting...
W20260621 07:09:32.564544 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:32 dist-test-slave-hlq1 krb5kdc[23504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:32.866324 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41177
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:33.253245 23517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:33.253618 23517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:33.253820 23517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:33.261731 23517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:33.261864 23517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:33.261987 23517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:33.262087 23517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:33.262173 23517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:33.273128 23517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:33.274832 23517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:33.276543 23517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:33.289585 23522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:33.289680 23523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:33.289584 23525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:33.290727 23517 server_base.cc:1034] running on GCE node
I20260621 07:09:33.291713 23517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:33.293411 23517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:33.294667 23517 hybrid_clock.cc:648] HybridClock initialized: now 1782025773294614 us; error 46 us; skew 500 ppm
I20260621 07:09:33.295086 23517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:33 dist-test-slave-hlq1 krb5kdc[23504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025773, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:33.304584 23517 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:33.306342 23517 webserver.cc:460] Webserver started at http://127.1.188.254:35787/ using document root <none> and password file <none>
I20260621 07:09:33.307183 23517 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:33.307325 23517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:33.307740 23517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:33.311266 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9666511852254d8b8c4b604a01c7cbef"
format_stamp: "Formatted at 2026-06-21 07:09:33 on dist-test-slave-hlq1"
server_key: "47cdaf4e67063266b88d4cfb9dc699a5"
server_key_iv: "091b69ded31c40119fef0502191a9bb9"
server_key_version: "encryptionkey@0"
I20260621 07:09:33.312234 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9666511852254d8b8c4b604a01c7cbef"
format_stamp: "Formatted at 2026-06-21 07:09:33 on dist-test-slave-hlq1"
server_key: "47cdaf4e67063266b88d4cfb9dc699a5"
server_key_iv: "091b69ded31c40119fef0502191a9bb9"
server_key_version: "encryptionkey@0"
I20260621 07:09:33.318239 23517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:09:33.322139 23532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:33.323665 23517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:09:33.323868 23517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9666511852254d8b8c4b604a01c7cbef"
format_stamp: "Formatted at 2026-06-21 07:09:33 on dist-test-slave-hlq1"
server_key: "47cdaf4e67063266b88d4cfb9dc699a5"
server_key_iv: "091b69ded31c40119fef0502191a9bb9"
server_key_version: "encryptionkey@0"
I20260621 07:09:33.324084 23517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:33.345726 23517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:33.350580 23517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:33.350914 23517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:33.371954 23517 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41177
I20260621 07:09:33.371963 23583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41177 every 8 connection(s)
I20260621 07:09:33.373678 23517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:33.377965 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23517
I20260621 07:09:33.378177 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:33.378247 23584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:33.378711 1779 external_mini_cluster.cc:1428] Setting key 6de785644d2c184c92a766d1b7ecb38f
I20260621 07:09:33.394546 23584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef: Bootstrap starting.
Jun 21 07:09:33 dist-test-slave-hlq1 krb5kdc[23504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:33.398789 23584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:33.400152 23584 log.cc:824] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:33.403487 23584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef: No bootstrap required, opened a new log
I20260621 07:09:33.404362 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:33.380781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42069 (local address 127.1.188.254:41177)
0621 07:09:33.381719 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:33.381736 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:33.381795 (+ 59us) server_negotiation.cc:407] Connection header received
0621 07:09:33.383879 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:33.383911 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:33.384468 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:33.384974 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:33.386407 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:33.388510 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:33.389876 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:33.391046 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:33.391254 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:33.396221 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:33.396230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:33.396245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:33.396285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:33.399725 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:33.400686 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:33.400693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:33.400699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:33.400882 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:33.401392 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:33.401396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:33.401398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:33.401954 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:33.402293 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:33.402812 (+ 519us) server_negotiation.cc:299] Negotiation successful
0621 07:09:33.403129 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":382,"threads_started":1}
I20260621 07:09:33.408563 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:33.412213 23584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } }
I20260621 07:09:33.412622 23584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:33.412727 23584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9666511852254d8b8c4b604a01c7cbef, State: Initialized, Role: FOLLOWER
I20260621 07:09:33.413530 23584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } }
I20260621 07:09:33.413872 23584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:33.414041 23584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:33.414240 23584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:33.416747 23584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } }
I20260621 07:09:33.417344 23584 leader_election.cc:302] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9666511852254d8b8c4b604a01c7cbef; no voters:
I20260621 07:09:33.417982 23584 leader_election.cc:288] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:33.418125 23594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:33.419431 23594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [term 1 LEADER]: Becoming Leader. State: Replica: 9666511852254d8b8c4b604a01c7cbef, State: Running, Role: LEADER
I20260621 07:09:33.420184 23594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } }
I20260621 07:09:33.420527 23584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:33.429236 23595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9666511852254d8b8c4b604a01c7cbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } } }
I20260621 07:09:33.429877 23595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:33.430600 23596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9666511852254d8b8c4b604a01c7cbef. Latest consensus state: current_term: 1 leader_uuid: "9666511852254d8b8c4b604a01c7cbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9666511852254d8b8c4b604a01c7cbef" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41177 } } }
I20260621 07:09:33.430889 23596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:33.434302 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:33.413299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47721 (local address 127.1.188.254:41177)
0621 07:09:33.413477 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:33.413481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:33.414103 (+ 622us) server_negotiation.cc:407] Connection header received
0621 07:09:33.414292 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:33.414298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:33.414386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:33.414485 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:33.416148 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:33.416971 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:33.418635 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:33.419718 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:33.419832 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:33.423847 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:33.423858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:33.423866 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:33.423923 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:33.428790 (+ 4867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:33.430453 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:33.430462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:33.430468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:33.430751 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:33.431506 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:33.431514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:33.431521 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:33.432113 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:33.432492 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:33.433426 (+ 934us) server_negotiation.cc:299] Negotiation successful
0621 07:09:33.433762 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 07:09:33.440644 23602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:33.446380 23602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:33.456332 23602 catalog_manager.cc:1347] Generated new cluster ID: 4b69503b21fd4a9baf7cddbdd00fb1b5
I20260621 07:09:33.456459 23602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:33.486992 23602 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:33.488420 23602 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:33.498375 23602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9666511852254d8b8c4b604a01c7cbef: Generated new TSK 0
I20260621 07:09:33.499269 23602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:33.509388 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23517
I20260621 07:09:33.511200 23579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:33.606096 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23517
W20260621 07:09:33.625867 23502 connection.cc:535] client connection to 127.1.188.254:41177 recv error: Network error: failed to read from TLS socket (remote: 127.1.188.254:41177): Connection reset by peer (error 104)
2026-06-21T07:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:33 dist-test-slave-hlq1 krb5kdc[23616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:33 dist-test-slave-hlq1 krb5kdc[23616](info): set up 2 sockets
Jun 21 07:09:33 dist-test-slave-hlq1 krb5kdc[23616](info): commencing operation
krb5kdc: starting...
W20260621 07:09:34.712049 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:34 dist-test-slave-hlq1 krb5kdc[23616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:34.968935 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40941
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:46471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:35.326284 23629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:35.326644 23629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:35.326833 23629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:35.334160 23629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:35.334317 23629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:35.334427 23629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:35.334522 23629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:35.334589 23629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:35.344403 23629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:46471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:35.345829 23629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:35.347194 23629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:35.358851 23634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:35.358909 23637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:35.358932 23635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:35.360013 23629 server_base.cc:1034] running on GCE node
I20260621 07:09:35.360785 23629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:35.362128 23629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:35.363361 23629 hybrid_clock.cc:648] HybridClock initialized: now 1782025775363285 us; error 57 us; skew 500 ppm
I20260621 07:09:35.363772 23629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:35 dist-test-slave-hlq1 krb5kdc[23616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025775, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:35.366892 23629 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:35.368410 23629 webserver.cc:460] Webserver started at http://127.1.188.254:37529/ using document root <none> and password file <none>
I20260621 07:09:35.369086 23629 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:35.369236 23629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:35.369534 23629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:35.372555 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "3601a5825fef4ef893c18d69774e7a53"
format_stamp: "Formatted at 2026-06-21 07:09:35 on dist-test-slave-hlq1"
server_key: "b832295326d1405d160452bea4503f7e"
server_key_iv: "80bf60e6a62db400ac8ef5a0f5648ae4"
server_key_version: "encryptionkey@0"
I20260621 07:09:35.373308 23629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "3601a5825fef4ef893c18d69774e7a53"
format_stamp: "Formatted at 2026-06-21 07:09:35 on dist-test-slave-hlq1"
server_key: "b832295326d1405d160452bea4503f7e"
server_key_iv: "80bf60e6a62db400ac8ef5a0f5648ae4"
server_key_version: "encryptionkey@0"
I20260621 07:09:35.378398 23629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:09:35.381848 23644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:35.383102 23629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 07:09:35.383286 23629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "3601a5825fef4ef893c18d69774e7a53"
format_stamp: "Formatted at 2026-06-21 07:09:35 on dist-test-slave-hlq1"
server_key: "b832295326d1405d160452bea4503f7e"
server_key_iv: "80bf60e6a62db400ac8ef5a0f5648ae4"
server_key_version: "encryptionkey@0"
I20260621 07:09:35.383546 23629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:35.411974 23629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:35.416227 23629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:35.416525 23629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:35.435261 23629 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40941
I20260621 07:09:35.435316 23695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40941 every 8 connection(s)
I20260621 07:09:35.436681 23629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:35.440660 23696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:35.445003 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23629
I20260621 07:09:35.445268 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:35.445741 1779 external_mini_cluster.cc:1428] Setting key 921803790cfb6a773c2e78948e7a1554
I20260621 07:09:35.453215 23696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53: Bootstrap starting.
I20260621 07:09:35.457057 23696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:35.458307 23696 log.cc:824] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:35.461249 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:35.447640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55067 (local address 127.1.188.254:40941)
0621 07:09:35.448485 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:35.448500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:35.448550 (+ 50us) server_negotiation.cc:407] Connection header received
0621 07:09:35.451301 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:35.451331 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:35.451867 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:35.452301 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:35.452817 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:35.454840 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:35.456275 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:35.457351 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:35.457565 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:35.458738 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:35.458748 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:35.458768 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:35.459574 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:35.459660 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:35.459686 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:35.459877 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:35.460142 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 07:09:35.460326 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":339,"threads_started":1}
I20260621 07:09:35.461602 23696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53: No bootstrap required, opened a new log
I20260621 07:09:35.465335 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:35.469982 23696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } }
I20260621 07:09:35.470489 23696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:35.470616 23696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3601a5825fef4ef893c18d69774e7a53, State: Initialized, Role: FOLLOWER
I20260621 07:09:35.471298 23696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } }
I20260621 07:09:35.471621 23696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:35.471794 23696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:35.472009 23696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:35.474279 23696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } }
I20260621 07:09:35.474817 23696 leader_election.cc:302] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3601a5825fef4ef893c18d69774e7a53; no voters:
I20260621 07:09:35.475550 23696 leader_election.cc:288] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:35.475651 23706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:35.476533 23706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [term 1 LEADER]: Becoming Leader. State: Replica: 3601a5825fef4ef893c18d69774e7a53, State: Running, Role: LEADER
I20260621 07:09:35.477247 23706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } }
I20260621 07:09:35.477483 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:35.470059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51617 (local address 127.1.188.254:40941)
0621 07:09:35.470255 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:35.470262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:35.470670 (+ 408us) server_negotiation.cc:407] Connection header received
0621 07:09:35.470859 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:35.470863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:35.470928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:35.471042 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:35.471595 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:35.472518 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:35.474319 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:35.474959 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:35.475117 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:35.476381 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:35.476389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:35.476398 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:35.476423 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:35.476481 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:35.476508 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:35.476639 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:35.477041 (+ 402us) server_negotiation.cc:299] Negotiation successful
0621 07:09:35.477101 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:09:35.478615 23696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:35.480605 23707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3601a5825fef4ef893c18d69774e7a53. Latest consensus state: current_term: 1 leader_uuid: "3601a5825fef4ef893c18d69774e7a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } } }
I20260621 07:09:35.480876 23708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3601a5825fef4ef893c18d69774e7a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3601a5825fef4ef893c18d69774e7a53" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40941 } } }
I20260621 07:09:35.481086 23707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:35.481154 23708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:35.484254 23713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:35.489418 23713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:35.499032 23713 catalog_manager.cc:1347] Generated new cluster ID: 26b9b8113375485a86f65c26e7922f94
I20260621 07:09:35.499250 23713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:35.510656 23713 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:35.512596 23713 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:35.530910 23713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3601a5825fef4ef893c18d69774e7a53: Generated new TSK 0
I20260621 07:09:35.532091 23713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:35.546933 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23629
W20260621 07:09:35.548810 23691 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55067
W20260621 07:09:35.549587 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23629: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:35.549742 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23629
W20260621 07:09:35.566839 23614 connection.cc:535] client connection to 127.1.188.254:40941 recv error: Network error: failed to read from TLS socket (remote: 127.1.188.254:40941): Connection reset by peer (error 104)
2026-06-21T07:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:35 dist-test-slave-hlq1 krb5kdc[23727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:35 dist-test-slave-hlq1 krb5kdc[23727](info): set up 2 sockets
Jun 21 07:09:35 dist-test-slave-hlq1 krb5kdc[23727](info): commencing operation
krb5kdc: starting...
W20260621 07:09:36.672538 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:36 dist-test-slave-hlq1 krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:36.981014 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40583
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:37.369935 23740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:37.370354 23740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:37.370573 23740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:37.378222 23740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:37.378355 23740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:37.378533 23740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:37.378641 23740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:37.378732 23740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:37.389379 23740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:37.391018 23740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:37.392620 23740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:37.406013 23745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:37.406268 23748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:37.406275 23746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:37.407022 23740 server_base.cc:1034] running on GCE node
I20260621 07:09:37.407990 23740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:37.409677 23740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:37.410916 23740 hybrid_clock.cc:648] HybridClock initialized: now 1782025777410863 us; error 48 us; skew 500 ppm
I20260621 07:09:37.411335 23740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:37 dist-test-slave-hlq1 krb5kdc[23727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025777, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:37.414970 23740 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:37.416841 23740 webserver.cc:460] Webserver started at http://127.1.188.254:38839/ using document root <none> and password file <none>
I20260621 07:09:37.417673 23740 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:37.417830 23740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:37.418190 23740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:37.421711 23740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "853028d06c9349cfaf1feb69aae0f2a3"
format_stamp: "Formatted at 2026-06-21 07:09:37 on dist-test-slave-hlq1"
server_key: "dd7922caa50ffea19e5e349ff85b3583"
server_key_iv: "4ef0be7c991f444a86de4b1ede8f2cf8"
server_key_version: "encryptionkey@0"
I20260621 07:09:37.422578 23740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "853028d06c9349cfaf1feb69aae0f2a3"
format_stamp: "Formatted at 2026-06-21 07:09:37 on dist-test-slave-hlq1"
server_key: "dd7922caa50ffea19e5e349ff85b3583"
server_key_iv: "4ef0be7c991f444a86de4b1ede8f2cf8"
server_key_version: "encryptionkey@0"
I20260621 07:09:37.428599 23740 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260621 07:09:37.432433 23755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:37.433944 23740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:37.434170 23740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "853028d06c9349cfaf1feb69aae0f2a3"
format_stamp: "Formatted at 2026-06-21 07:09:37 on dist-test-slave-hlq1"
server_key: "dd7922caa50ffea19e5e349ff85b3583"
server_key_iv: "4ef0be7c991f444a86de4b1ede8f2cf8"
server_key_version: "encryptionkey@0"
I20260621 07:09:37.434365 23740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:37.440671 23740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:37.445140 23740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:37.445398 23740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:37.466217 23740 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40583
I20260621 07:09:37.466217 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40583 every 8 connection(s)
I20260621 07:09:37.468164 23740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:37.473165 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:37.475257 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23740
I20260621 07:09:37.475508 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:37.476184 1779 external_mini_cluster.cc:1428] Setting key f75308e08f25d48bb4741eb5d2711fa9
I20260621 07:09:37.485122 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:37.478372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50231 (local address 127.1.188.254:40583)
0621 07:09:37.479347 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:37.479365 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:37.479495 (+ 130us) server_negotiation.cc:407] Connection header received
0621 07:09:37.481269 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:37.481296 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:37.481717 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:37.482100 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:37.483293 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:37.483300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:37.483314 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:37.483328 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:37.483394 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:37.483473 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:37.483670 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:37.483773 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 07:09:37.484006 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":482,"threads_started":1}
I20260621 07:09:37.488626 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3: Bootstrap starting.
I20260621 07:09:37.489296 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:37.492357 23807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:37.493695 23807 log.cc:824] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:37.497216 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:37.494314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49493 (local address 127.1.188.254:40583)
0621 07:09:37.494559 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:37.494569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:37.495068 (+ 499us) server_negotiation.cc:407] Connection header received
0621 07:09:37.495285 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:37.495291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:37.495350 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:37.495510 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:37.496532 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:37.496539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:37.496556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:37.496578 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:37.496631 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:37.496652 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:37.496771 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:37.496849 (+ 78us) server_negotiation.cc:299] Negotiation successful
0621 07:09:37.496919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:09:37.497556 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3: No bootstrap required, opened a new log
I20260621 07:09:37.506889 23807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } }
I20260621 07:09:37.507414 23807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:37.507582 23807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 853028d06c9349cfaf1feb69aae0f2a3, State: Initialized, Role: FOLLOWER
I20260621 07:09:37.508303 23807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } }
I20260621 07:09:37.508572 23807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:37.508750 23807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:37.508941 23807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:37.511571 23807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } }
I20260621 07:09:37.512220 23807 leader_election.cc:302] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 853028d06c9349cfaf1feb69aae0f2a3; no voters:
I20260621 07:09:37.512840 23807 leader_election.cc:288] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:37.513012 23817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:37.513813 23817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [term 1 LEADER]: Becoming Leader. State: Replica: 853028d06c9349cfaf1feb69aae0f2a3, State: Running, Role: LEADER
I20260621 07:09:37.514449 23817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } }
I20260621 07:09:37.515328 23807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:37.517704 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "853028d06c9349cfaf1feb69aae0f2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } } }
I20260621 07:09:37.517717 23819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 853028d06c9349cfaf1feb69aae0f2a3. Latest consensus state: current_term: 1 leader_uuid: "853028d06c9349cfaf1feb69aae0f2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "853028d06c9349cfaf1feb69aae0f2a3" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40583 } } }
I20260621 07:09:37.519680 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:37.519729 23819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:37.522153 23823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:37.528541 23823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:37.538318 23823 catalog_manager.cc:1347] Generated new cluster ID: d5f852f025354f06b16316c803b32986
I20260621 07:09:37.538458 23823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:37.556886 23823 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:37.557333 23823 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:37.566318 23823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 853028d06c9349cfaf1feb69aae0f2a3: Generated new TSK 0
I20260621 07:09:37.567363 23823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:37.583773 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23740
W20260621 07:09:37.584860 23802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50231
W20260621 07:09:37.585583 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23740: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:37.585734 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23740
2026-06-21T07:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:37 dist-test-slave-hlq1 krb5kdc[23838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:37 dist-test-slave-hlq1 krb5kdc[23838](info): set up 2 sockets
Jun 21 07:09:37 dist-test-slave-hlq1 krb5kdc[23838](info): commencing operation
krb5kdc: starting...
W20260621 07:09:38.704406 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:38 dist-test-slave-hlq1 krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:39.014619 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40369
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:39.417933 23851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:39.418315 23851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:39.418503 23851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:39.426378 23851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:39.426565 23851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:39.426672 23851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:39.426749 23851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:39.426797 23851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:39.437938 23851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:39.439635 23851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:39.441222 23851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:39.454432 23856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:39.454389 23857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:39.454481 23859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:39.455309 23851 server_base.cc:1034] running on GCE node
I20260621 07:09:39.456278 23851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:39.458086 23851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:39.459335 23851 hybrid_clock.cc:648] HybridClock initialized: now 1782025779459279 us; error 49 us; skew 500 ppm
I20260621 07:09:39.459779 23851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:39 dist-test-slave-hlq1 krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025779, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:39.467207 23851 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:39.469220 23851 webserver.cc:460] Webserver started at http://127.1.188.254:33881/ using document root <none> and password file <none>
I20260621 07:09:39.470096 23851 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:39.470265 23851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:39.470635 23851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:39.474340 23851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "8bef0555ccf4467d8fbff03b5820b260"
format_stamp: "Formatted at 2026-06-21 07:09:39 on dist-test-slave-hlq1"
server_key: "58f4f9b0b5ec993fd1409d8f272c4024"
server_key_iv: "2e7db0dc0d42646acd661dd0d454576b"
server_key_version: "encryptionkey@0"
I20260621 07:09:39.475253 23851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "8bef0555ccf4467d8fbff03b5820b260"
format_stamp: "Formatted at 2026-06-21 07:09:39 on dist-test-slave-hlq1"
server_key: "58f4f9b0b5ec993fd1409d8f272c4024"
server_key_iv: "2e7db0dc0d42646acd661dd0d454576b"
server_key_version: "encryptionkey@0"
I20260621 07:09:39.481472 23851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260621 07:09:39.485599 23866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:39.487135 23851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:09:39.487344 23851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "8bef0555ccf4467d8fbff03b5820b260"
format_stamp: "Formatted at 2026-06-21 07:09:39 on dist-test-slave-hlq1"
server_key: "58f4f9b0b5ec993fd1409d8f272c4024"
server_key_iv: "2e7db0dc0d42646acd661dd0d454576b"
server_key_version: "encryptionkey@0"
I20260621 07:09:39.487679 23851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:39.511277 23851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:39.516283 23851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:39.516631 23851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:39.538511 23851 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40369
I20260621 07:09:39.538514 23917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40369 every 8 connection(s)
I20260621 07:09:39.540305 23851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:39.544853 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:39.549364 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23851
I20260621 07:09:39.549608 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:39.550172 1779 external_mini_cluster.cc:1428] Setting key 72ded39a9fc6b315fb6ab7a50d066a0e
I20260621 07:09:39.558663 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260: Bootstrap starting.
I20260621 07:09:39.562942 23918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:39.564500 23918 log.cc:824] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260: Log is configured to *not* fsync() on all Append() calls
Jun 21 07:09:39 dist-test-slave-hlq1 krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:39.568337 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260: No bootstrap required, opened a new log
I20260621 07:09:39.580595 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:39.552462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38519 (local address 127.1.188.254:40369)
0621 07:09:39.553408 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:39.553427 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:39.553487 (+ 60us) server_negotiation.cc:407] Connection header received
0621 07:09:39.556614 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:39.556648 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:39.557261 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:39.557776 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:39.559345 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:39.561517 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:39.562894 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:39.564045 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:39.564260 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:39.569402 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:39.569412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:39.569433 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:39.569479 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:39.574614 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:39.575791 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:39.575801 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:39.575812 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:39.576072 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:39.576574 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:39.576580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:39.576584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:39.577523 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:39.577872 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:39.578283 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 07:09:39.578861 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":408,"threads_started":1}
I20260621 07:09:39.581301 23918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } }
I20260621 07:09:39.581750 23918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:39.581880 23918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bef0555ccf4467d8fbff03b5820b260, State: Initialized, Role: FOLLOWER
I20260621 07:09:39.582679 23918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } }
I20260621 07:09:39.583029 23918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:39.583180 23918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:39.583387 23918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:39.585400 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:39.587009 23918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } }
I20260621 07:09:39.587662 23918 leader_election.cc:302] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bef0555ccf4467d8fbff03b5820b260; no voters:
I20260621 07:09:39.588338 23918 leader_election.cc:288] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:39.588590 23923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:39.591012 23918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:39.591683 23923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [term 1 LEADER]: Becoming Leader. State: Replica: 8bef0555ccf4467d8fbff03b5820b260, State: Running, Role: LEADER
I20260621 07:09:39.592528 23923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } }
I20260621 07:09:39.596276 23925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bef0555ccf4467d8fbff03b5820b260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } } }
I20260621 07:09:39.596318 23926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bef0555ccf4467d8fbff03b5820b260. Latest consensus state: current_term: 1 leader_uuid: "8bef0555ccf4467d8fbff03b5820b260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef0555ccf4467d8fbff03b5820b260" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40369 } } }
I20260621 07:09:39.596678 23925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:39.596721 23926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:39.597649 23932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:39.608755 23932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:39.630080 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:39.596782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46419 (local address 127.1.188.254:40369)
0621 07:09:39.597005 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:39.597012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:39.604696 (+ 7684us) server_negotiation.cc:407] Connection header received
0621 07:09:39.604944 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:39.604952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:39.605048 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:39.605135 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:39.607352 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:39.608575 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:39.610804 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:39.611559 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:39.611674 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:39.615310 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:39.615317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:39.615323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:39.615363 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:39.619248 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:39.624794 (+ 5546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:39.624799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:39.624803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:39.625179 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:39.626467 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:39.626475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:39.626478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:39.626859 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:39.629334 (+ 2475us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:39.629486 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 07:09:39.629705 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 07:09:39.632367 23932 catalog_manager.cc:1347] Generated new cluster ID: 5d0a49415fec46cabadfca857e126839
I20260621 07:09:39.632501 23932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:39.648973 23932 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:39.650774 23932 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:39.664160 23932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bef0555ccf4467d8fbff03b5820b260: Generated new TSK 0
I20260621 07:09:39.665037 23932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:39.674115 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23851
I20260621 07:09:39.675382 23913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:39.768441 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23851
2026-06-21T07:09:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2181 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:39 dist-test-slave-hlq1 krb5kdc[23950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:39 dist-test-slave-hlq1 krb5kdc[23950](info): set up 2 sockets
Jun 21 07:09:39 dist-test-slave-hlq1 krb5kdc[23950](info): commencing operation
krb5kdc: starting...
W20260621 07:09:40.876178 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:41 dist-test-slave-hlq1 krb5kdc[23950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:41.135046 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:34765
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:41.484187 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:41.484517 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:41.484687 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:41.492148 23963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:41.492252 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:41.492367 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:41.492766 23963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:41.492853 23963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:41.503288 23963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:41.504805 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:41.506165 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:41.518321 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:41.518378 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:41.518404 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:41.519309 23963 server_base.cc:1034] running on GCE node
I20260621 07:09:41.520149 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:41.521518 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:41.522748 23963 hybrid_clock.cc:648] HybridClock initialized: now 1782025781522713 us; error 50 us; skew 500 ppm
I20260621 07:09:41.523090 23963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:41 dist-test-slave-hlq1 krb5kdc[23950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025781, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:41.526178 23963 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:41.527773 23963 webserver.cc:460] Webserver started at http://127.1.188.254:38913/ using document root <none> and password file <none>
I20260621 07:09:41.528469 23963 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:41.528617 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:41.528932 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:41.531950 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "eba3f0a655194ea9b6254f9aee3b84cc"
format_stamp: "Formatted at 2026-06-21 07:09:41 on dist-test-slave-hlq1"
server_key: "98ccdc0881ac810cd24d04c3f6656b40"
server_key_iv: "0fef285ec4bd44f193283a46e35bbf03"
server_key_version: "encryptionkey@0"
I20260621 07:09:41.532718 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "eba3f0a655194ea9b6254f9aee3b84cc"
format_stamp: "Formatted at 2026-06-21 07:09:41 on dist-test-slave-hlq1"
server_key: "98ccdc0881ac810cd24d04c3f6656b40"
server_key_iv: "0fef285ec4bd44f193283a46e35bbf03"
server_key_version: "encryptionkey@0"
I20260621 07:09:41.538101 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 07:09:41.541699 23978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:41.542965 23963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 07:09:41.543154 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "eba3f0a655194ea9b6254f9aee3b84cc"
format_stamp: "Formatted at 2026-06-21 07:09:41 on dist-test-slave-hlq1"
server_key: "98ccdc0881ac810cd24d04c3f6656b40"
server_key_iv: "0fef285ec4bd44f193283a46e35bbf03"
server_key_version: "encryptionkey@0"
I20260621 07:09:41.543355 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:41.561108 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:41.565043 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:41.565368 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:41.583829 23963 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:34765
I20260621 07:09:41.583853 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:34765 every 8 connection(s)
I20260621 07:09:41.585157 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:41.588778 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:41.590129 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 23963
I20260621 07:09:41.590318 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:41.590915 1779 external_mini_cluster.cc:1428] Setting key b2e6f622ab86ab26f8672ee9dc4f416a
I20260621 07:09:41.603454 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc: Bootstrap starting.
I20260621 07:09:41.604454 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:41.592750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38105 (local address 127.1.188.254:34765)
0621 07:09:41.593517 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:41.593531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:41.593570 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:09:41.595165 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:41.595189 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:41.595635 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:41.595962 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:41.596504 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:41.597756 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:41.599968 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:41.600665 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:41.600816 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:41.602233 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:41.602243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:41.602257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:41.602786 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:41.602858 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:41.602883 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:41.602990 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:41.603214 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 07:09:41.603503 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":377,"threads_started":1}
I20260621 07:09:41.607157 24030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:41.608361 24030 log.cc:824] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:41.608552 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:41.611099 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc: No bootstrap required, opened a new log
I20260621 07:09:41.619557 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:41.613273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52179 (local address 127.1.188.254:34765)
0621 07:09:41.613491 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:41.613500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:41.613899 (+ 399us) server_negotiation.cc:407] Connection header received
0621 07:09:41.614106 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:41.614111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:41.614171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:41.614283 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:41.614783 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:41.615569 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:41.616890 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:41.617338 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:41.617487 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:41.618622 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:41.618629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:41.618633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:41.618656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:41.618715 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:41.618741 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:41.618839 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:41.619019 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 07:09:41.619126 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 07:09:41.619727 24030 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } }
I20260621 07:09:41.620265 24030 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:41.620386 24030 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba3f0a655194ea9b6254f9aee3b84cc, State: Initialized, Role: FOLLOWER
I20260621 07:09:41.621037 24030 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } }
I20260621 07:09:41.621374 24030 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:41.621531 24030 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:41.621708 24030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:41.624163 24030 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } }
I20260621 07:09:41.624740 24030 leader_election.cc:302] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba3f0a655194ea9b6254f9aee3b84cc; no voters:
I20260621 07:09:41.625607 24040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:41.626098 24030 leader_election.cc:288] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:41.626194 24040 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [term 1 LEADER]: Becoming Leader. State: Replica: eba3f0a655194ea9b6254f9aee3b84cc, State: Running, Role: LEADER
I20260621 07:09:41.626776 24040 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } }
I20260621 07:09:41.629580 24030 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:41.630514 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } } }
I20260621 07:09:41.630923 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:41.633275 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba3f0a655194ea9b6254f9aee3b84cc. Latest consensus state: current_term: 1 leader_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba3f0a655194ea9b6254f9aee3b84cc" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 34765 } } }
I20260621 07:09:41.633625 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:41.634549 24047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:41.639729 24047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:41.649272 24047 catalog_manager.cc:1347] Generated new cluster ID: 6504394424ca4b0da40b812766462f6c
I20260621 07:09:41.649381 24047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:41.666428 24047 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:41.668394 24047 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:41.681829 24047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba3f0a655194ea9b6254f9aee3b84cc: Generated new TSK 0
I20260621 07:09:41.682754 24047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:41.690907 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23963
W20260621 07:09:41.692039 24025 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38105
W20260621 07:09:41.692749 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 23963: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:41.692909 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 23963
2026-06-21T07:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:41 dist-test-slave-hlq1 krb5kdc[24061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:41 dist-test-slave-hlq1 krb5kdc[24061](info): set up 2 sockets
Jun 21 07:09:41 dist-test-slave-hlq1 krb5kdc[24061](info): commencing operation
krb5kdc: starting...
W20260621 07:09:42.808094 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:42 dist-test-slave-hlq1 krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:43.064944 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40125
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:43.390497 24074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:43.390758 24074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:43.390848 24074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:43.397835 24074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:43.397962 24074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:43.398010 24074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:43.398073 24074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:43.398116 24074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:43.407923 24074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:43.409125 24074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:43.410362 24074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:43.421965 24080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:43.422060 24079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:43.422022 24082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:43.423056 24074 server_base.cc:1034] running on GCE node
I20260621 07:09:43.423939 24074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:43.425311 24074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:43.426509 24074 hybrid_clock.cc:648] HybridClock initialized: now 1782025783426469 us; error 36 us; skew 500 ppm
I20260621 07:09:43.426801 24074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:43 dist-test-slave-hlq1 krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025783, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:43.429903 24074 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:43.431346 24074 webserver.cc:460] Webserver started at http://127.1.188.254:37117/ using document root <none> and password file <none>
I20260621 07:09:43.432113 24074 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:43.432263 24074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:43.432564 24074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:43.435463 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "b08530fea7a645adacb89160c1700f12"
format_stamp: "Formatted at 2026-06-21 07:09:43 on dist-test-slave-hlq1"
server_key: "8ee4fb8adb29c6cfed67c42525c52514"
server_key_iv: "4681bd0ecd5651d684ba2420a0147e43"
server_key_version: "encryptionkey@0"
I20260621 07:09:43.436244 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "b08530fea7a645adacb89160c1700f12"
format_stamp: "Formatted at 2026-06-21 07:09:43 on dist-test-slave-hlq1"
server_key: "8ee4fb8adb29c6cfed67c42525c52514"
server_key_iv: "4681bd0ecd5651d684ba2420a0147e43"
server_key_version: "encryptionkey@0"
I20260621 07:09:43.441432 24074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260621 07:09:43.444952 24089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:43.446382 24074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 07:09:43.446563 24074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "b08530fea7a645adacb89160c1700f12"
format_stamp: "Formatted at 2026-06-21 07:09:43 on dist-test-slave-hlq1"
server_key: "8ee4fb8adb29c6cfed67c42525c52514"
server_key_iv: "4681bd0ecd5651d684ba2420a0147e43"
server_key_version: "encryptionkey@0"
I20260621 07:09:43.446763 24074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:43.452577 24074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:43.456565 24074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:43.456868 24074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:43.476056 24074 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:40125
I20260621 07:09:43.476102 24140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:40125 every 8 connection(s)
I20260621 07:09:43.477507 24074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:43.479315 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24074
I20260621 07:09:43.479534 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:43.480001 1779 external_mini_cluster.cc:1428] Setting key a4ced1a0f103ece5c74dee0f0fef0f3e
I20260621 07:09:43.481825 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:43.487655 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:43.481695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47739 (local address 127.1.188.254:40125)
0621 07:09:43.482519 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:43.482530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:43.482569 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:09:43.484212 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:43.484231 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:43.484549 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:43.484884 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:43.485976 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:43.485982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:43.485993 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:43.486007 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:43.486079 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:43.486098 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:43.486259 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:43.486386 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 07:09:43.486594 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":409,"threads_started":1}
I20260621 07:09:43.491127 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:43.494889 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12: Bootstrap starting.
I20260621 07:09:43.499023 24141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:43.499317 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:43.496490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50255 (local address 127.1.188.254:40125)
0621 07:09:43.496705 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:43.496711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:43.497063 (+ 352us) server_negotiation.cc:407] Connection header received
0621 07:09:43.497329 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:43.497333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:43.497385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:43.497479 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:43.498479 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:43.498483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:43.498486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 07:09:43.498501 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:43.498540 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 07:09:43.498555 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:43.498648 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:43.498852 (+ 204us) server_negotiation.cc:299] Negotiation successful
0621 07:09:43.498907 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 07:09:43.500346 24141 log.cc:824] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:43.503103 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12: No bootstrap required, opened a new log
I20260621 07:09:43.510434 24141 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } }
I20260621 07:09:43.510859 24141 raft_consensus.cc:374] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 07:09:43.510968 24141 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08530fea7a645adacb89160c1700f12, State: Initialized, Role: FOLLOWER
I20260621 07:09:43.511597 24141 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } }
I20260621 07:09:43.511848 24141 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 07:09:43.511996 24141 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 07:09:43.512171 24141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:43.514351 24141 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } }
I20260621 07:09:43.514954 24141 leader_election.cc:302] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08530fea7a645adacb89160c1700f12; no voters:
I20260621 07:09:43.515638 24141 leader_election.cc:288] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 07:09:43.515772 24151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:43.516481 24151 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [term 1 LEADER]: Becoming Leader. State: Replica: b08530fea7a645adacb89160c1700f12, State: Running, Role: LEADER
I20260621 07:09:43.516996 24151 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } }
I20260621 07:09:43.517724 24141 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:43.519955 24152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b08530fea7a645adacb89160c1700f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } } }
I20260621 07:09:43.520431 24152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:43.521719 24153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08530fea7a645adacb89160c1700f12. Latest consensus state: current_term: 1 leader_uuid: "b08530fea7a645adacb89160c1700f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08530fea7a645adacb89160c1700f12" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 40125 } } }
I20260621 07:09:43.522080 24153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12 [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:43.525936 24159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 07:09:43.530344 24159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:43.538612 24159 catalog_manager.cc:1347] Generated new cluster ID: dd398bf6c9224bcba1c04fa9fcd37ff5
I20260621 07:09:43.538753 24159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:43.561618 24159 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:43.562099 24159 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:43.573678 24159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b08530fea7a645adacb89160c1700f12: Generated new TSK 0
I20260621 07:09:43.574692 24159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:43.591985 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24074
W20260621 07:09:43.593050 24136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47739
W20260621 07:09:43.593798 1779 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 07:09:43.593947 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24074
2026-06-21T07:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1901 ms)
[----------] 18 tests from AuthTokenIssuingTest (36602 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:43 dist-test-slave-hlq1 krb5kdc[24172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:43 dist-test-slave-hlq1 krb5kdc[24172](info): set up 2 sockets
Jun 21 07:09:43 dist-test-slave-hlq1 krb5kdc[24172](info): commencing operation
krb5kdc: starting...
W20260621 07:09:44.708118 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:44 dist-test-slave-hlq1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:44.974443 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:43295
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:45.312835 24185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:45.313176 24185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:45.313305 24185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:45.313457 24185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:45.320576 24185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:45.320655 24185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:45.320760 24185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:45.320844 24185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:45.320935 24185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:45.330598 24185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:45.331954 24185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:45.333204 24185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:45.344923 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:45.344894 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:45.345034 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:45.345654 24185 server_base.cc:1034] running on GCE node
I20260621 07:09:45.346380 24185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:45.347739 24185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:45.348977 24185 hybrid_clock.cc:648] HybridClock initialized: now 1782025785348914 us; error 39 us; skew 500 ppm
I20260621 07:09:45.349293 24185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:45 dist-test-slave-hlq1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:45.356161 24185 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:45.357592 24185 webserver.cc:460] Webserver started at http://127.1.188.254:38553/ using document root <none> and password file <none>
I20260621 07:09:45.358227 24185 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:45.358312 24185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:45.358575 24185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:45.361471 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "9fda74d732cc40a7982ae5ee4505729b"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "587aabbc1068173665b482ebd9beeb3d"
server_key_iv: "135c7d11716555abd8ea5d37d11a66e5"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.362175 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "9fda74d732cc40a7982ae5ee4505729b"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "587aabbc1068173665b482ebd9beeb3d"
server_key_iv: "135c7d11716555abd8ea5d37d11a66e5"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.367116 24185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 07:09:45.370620 24200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:45.372051 24185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 07:09:45.372233 24185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "9fda74d732cc40a7982ae5ee4505729b"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "587aabbc1068173665b482ebd9beeb3d"
server_key_iv: "135c7d11716555abd8ea5d37d11a66e5"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.372452 24185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:45.422714 24185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:45.427147 24185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:45.427461 24185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:45.446050 24185 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:43295
I20260621 07:09:45.446069 24251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:43295 every 8 connection(s)
I20260621 07:09:45.447472 24185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:45.451098 24252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:45.451647 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24185
I20260621 07:09:45.451869 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:45.452687 1779 external_mini_cluster.cc:1428] Setting key 725081963a423d1c4f9ea8c1f394c117
I20260621 07:09:45.461247 24252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:45 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.253@KRBTEST.COM" created.
I20260621 07:09:45.486487 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.465190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.1.188.254:43295)
0621 07:09:45.466876 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:45.466886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:45.466944 (+ 58us) server_negotiation.cc:407] Connection header received
0621 07:09:45.468570 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:45.468581 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:45.468875 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:45.469096 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:45.470237 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:45.472204 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:45.473663 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:45.474633 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:45.474826 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:45.478581 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:45.478596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:45.478614 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:45.478659 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:45.482241 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:45.483020 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:45.483027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:45.483029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:45.483168 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:45.483698 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:45.483707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:45.483713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:45.484238 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:45.484399 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:45.485089 (+ 690us) server_negotiation.cc:299] Negotiation successful
0621 07:09:45.485398 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":1470,"spinlock_wait_cycles":94976,"thread_start_us":1254,"threads_started":1}
I20260621 07:09:45.486523 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.465200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:43295 (local address 127.0.0.1:52420)
0621 07:09:45.466209 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.466269 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:45.468203 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:45.469105 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:45.469111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:45.469692 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:45.470024 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:45.470040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:45.472365 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:45.472370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:45.473507 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:45.473520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:45.474881 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:45.474887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:45.475112 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:45.475947 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:45.475975 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:45.478326 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:45.482388 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:45.482399 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:45.482408 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:45.482896 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:45.483313 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:45.483317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:45.483320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:45.483581 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:45.484386 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:45.484397 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:45.484670 (+ 273us) client_negotiation.cc:769] Sending connection context
0621 07:09:45.484965 (+ 295us) client_negotiation.cc:240] Negotiation successful
0621 07:09:45.485590 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":133,"thread_start_us":415,"threads_started":1}
I20260621 07:09:45.489570 24252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:45.490759 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.490356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59302)
0621 07:09:45.490526 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.490629 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260621 07:09:45.491081 24201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
I20260621 07:09:45.492733 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.492384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59306)
0621 07:09:45.492524 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.492593 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 07:09:45.493067 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:34113: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
I20260621 07:09:45.515935 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:34113
--webserver_interface=127.1.188.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:09:45.542429 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } attempt: 1
I20260621 07:09:45.543730 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.543178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59314)
0621 07:09:45.543437 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.543587 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 07:09:45.544867 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.544406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59330)
0621 07:09:45.544569 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.544669 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:09:45.545082 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:34113: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
I20260621 07:09:45.613364 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } attempt: 2
I20260621 07:09:45.614795 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.614089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59342)
0621 07:09:45.614421 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.614540 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260621 07:09:45.616079 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.615627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59348)
0621 07:09:45.615811 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.615904 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 07:09:45.616336 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:34113: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
I20260621 07:09:45.707644 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } attempt: 3
I20260621 07:09:45.709069 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.708426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59356)
0621 07:09:45.708728 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.708863 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260621 07:09:45.710412 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.709892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59370)
0621 07:09:45.710080 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.710187 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 07:09:45.710669 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:34113: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
I20260621 07:09:45.854013 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } attempt: 4
I20260621 07:09:45.855801 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.854928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59384)
0621 07:09:45.855367 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.855563 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260621 07:09:45.857079 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:45.856586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59396)
0621 07:09:45.856773 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:45.856893 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 07:09:45.857347 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:34113: Network error: Client connection negotiation failed: client connection to 127.1.188.253:34113: connect: Connection refused (error 111)
W20260621 07:09:45.877202 24259 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:45.877584 24259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:45.877669 24259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:45.877766 24259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:45.885627 24259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:45.885739 24259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:45.885812 24259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:45.885875 24259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:45.885960 24259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:45.896880 24259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.188.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:45.898478 24259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:45.900051 24259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:45.913735 24267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:45.913774 24264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:45.913669 24265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:45.915390 24259 server_base.cc:1034] running on GCE node
I20260621 07:09:45.916361 24259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:45.918051 24259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:45.919323 24259 hybrid_clock.cc:648] HybridClock initialized: now 1782025785919255 us; error 52 us; skew 500 ppm
I20260621 07:09:45.919749 24259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:45 dist-test-slave-hlq1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:45.923470 24259 init.cc:377] Logged in from keytab as kudu/127.1.188.253@KRBTEST.COM (short username kudu)
I20260621 07:09:45.925151 24259 webserver.cc:460] Webserver started at http://127.1.188.253:43091/ using document root <none> and password file <none>
I20260621 07:09:45.925992 24259 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:45.926133 24259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:45.926504 24259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:45.930123 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data/instance:
uuid: "a27b5c09070e40b18940414d5192e573"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "78836ac8c4bb1d926599a34d64ea533f"
server_key_iv: "b66fa5132f53a70e5b439369223e8451"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.931123 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal/instance:
uuid: "a27b5c09070e40b18940414d5192e573"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "78836ac8c4bb1d926599a34d64ea533f"
server_key_iv: "b66fa5132f53a70e5b439369223e8451"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.937039 24259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:09:45.940899 24274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:45.942461 24259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:45.942667 24259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "a27b5c09070e40b18940414d5192e573"
format_stamp: "Formatted at 2026-06-21 07:09:45 on dist-test-slave-hlq1"
server_key: "78836ac8c4bb1d926599a34d64ea533f"
server_key_iv: "b66fa5132f53a70e5b439369223e8451"
server_key_version: "encryptionkey@0"
I20260621 07:09:45.942909 24259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:45.988639 24259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:45.993216 24259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:45.993537 24259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:46.014772 24259 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.253:34113
I20260621 07:09:46.014796 24325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.253:34113 every 8 connection(s)
I20260621 07:09:46.016435 24259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
I20260621 07:09:46.020931 24326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:46.022316 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24259
I20260621 07:09:46.022533 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-1/wal/instance
I20260621 07:09:46.023043 1779 external_mini_cluster.cc:1428] Setting key 52a940e2ee9137b84fb389674ec07915
I20260621 07:09:46.028472 24326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:46.054095 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.031609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.1.188.254:43295)
0621 07:09:46.032246 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.032256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.033224 (+ 968us) server_negotiation.cc:407] Connection header received
0621 07:09:46.036376 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.036382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.036462 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.036574 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.038858 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.039823 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.042261 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.042692 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.042787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.047548 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.047555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.047558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.047615 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.050719 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.051561 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.051568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.051573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.051711 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.052159 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.052164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.052166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.052477 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.052663 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.053421 (+ 758us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.053669 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.1.188.252@KRBTEST.COM; defaulting to no policy
I20260621 07:09:46.055294 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.032111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:43295 (local address 127.0.0.1:52436)
0621 07:09:46.033016 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.033072 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.036049 (+ 2977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.036848 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.036862 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.037704 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.038617 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.038644 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.039993 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.040002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.042114 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.042132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.042940 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.042950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.043297 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.044179 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.044216 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.047274 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.050869 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.050879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.050898 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.051392 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.051823 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.051829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.051833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.052025 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.052643 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.052651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.053028 (+ 377us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.053327 (+ 299us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.053909 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":392,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.252@KRBTEST.COM" created.
I20260621 07:09:46.057314 24326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:46.073167 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.058280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59408)
0621 07:09:46.058454 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.058472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.058566 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.059930 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.059935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.060391 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.060531 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.060539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.062010 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.062018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.062885 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.062896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.063821 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.063826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.064025 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.064664 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.064682 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.067075 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.070483 (+ 3408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.070489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.070494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.070854 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.071259 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.071265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.071269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.071468 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.072125 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.072130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.072266 (+ 136us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.072425 (+ 159us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.072740 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 07:09:46.073292 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.058585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.1.188.253:34113)
0621 07:09:46.059378 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.059385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.059491 (+ 106us) server_negotiation.cc:407] Connection header received
0621 07:09:46.059553 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.059560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.059766 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.059950 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.060714 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.061864 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.063024 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.063673 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.063813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.067291 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.067302 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.067317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.067345 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.070335 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.070989 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.071009 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.071012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.071126 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.071568 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.071576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.071579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.071952 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.072188 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.072579 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.072874 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":621,"thread_start_us":468,"threads_started":1}
I20260621 07:09:46.076298 24326 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:46.077701 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.077193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55818)
0621 07:09:46.077392 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.077531 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 07:09:46.078016 24277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.079777 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.079385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55822)
0621 07:09:46.079565 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.079652 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260621 07:09:46.080029 24326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
I20260621 07:09:46.094262 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:43035
--webserver_interface=127.1.188.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:09:46.129415 24326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 1
I20260621 07:09:46.131074 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.130360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55832)
0621 07:09:46.130610 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.130737 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 07:09:46.132548 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.132053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55840)
0621 07:09:46.132255 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.132398 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 07:09:46.132838 24326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.156672 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } attempt: 5
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:46.172446 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.157870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.1.188.253:34113)
0621 07:09:46.158125 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.158134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.158270 (+ 136us) server_negotiation.cc:407] Connection header received
0621 07:09:46.158557 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.158562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.158644 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.158765 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.159767 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.160531 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.161767 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.162287 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.162432 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.166373 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.166381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.166385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.166427 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.169456 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.170215 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.170222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.170226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.170367 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.170770 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.170777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.170780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.171106 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.171302 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.171785 (+ 483us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.172039 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 07:09:46.172442 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.157812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59420)
0621 07:09:46.158126 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.158150 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.158376 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.158816 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.158821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.159309 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.159589 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.159600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.160687 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.160693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.161600 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.161611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.162453 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.162458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.162686 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.163483 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.163505 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.166027 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.169612 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.169618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.169623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.170076 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.170483 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.170488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.170492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.170660 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.171253 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.171259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.171502 (+ 243us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.171669 (+ 167us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.172044 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260621 07:09:46.174068 24252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:46.175343 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.174868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55850)
0621 07:09:46.175087 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.175173 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 07:09:46.176466 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.176023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55864)
0621 07:09:46.176202 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.176301 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 07:09:46.176680 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.201138 24326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 2
I20260621 07:09:46.202665 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.202066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55880)
0621 07:09:46.202327 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.202457 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 07:09:46.203879 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.203512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55886)
0621 07:09:46.203667 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.203751 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:09:46.204147 24326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.227953 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 1
I20260621 07:09:46.229573 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.228884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55888)
0621 07:09:46.229147 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.229307 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260621 07:09:46.231125 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.230698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55894)
0621 07:09:46.230853 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.230956 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260621 07:09:46.231415 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.295480 24326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 3
I20260621 07:09:46.296938 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.296344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55896)
0621 07:09:46.296613 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.296732 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 07:09:46.298192 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.297701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55910)
0621 07:09:46.297966 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.298056 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260621 07:09:46.298439 24326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.299679 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 2
I20260621 07:09:46.300940 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.300359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55918)
0621 07:09:46.300603 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.300729 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 07:09:46.302053 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.301717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55930)
0621 07:09:46.301857 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.301931 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:09:46.302357 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.408620 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 3
I20260621 07:09:46.410115 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.409525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55938)
0621 07:09:46.409781 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.409894 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 07:09:46.411304 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.410852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55944)
0621 07:09:46.411083 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.411161 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260621 07:09:46.411614 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.441680 24326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 4
I20260621 07:09:46.443025 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.442478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55956)
0621 07:09:46.442702 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.442829 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 07:09:46.444238 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.443916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55968)
0621 07:09:46.444057 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.444134 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 07:09:46.444530 24326 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
W20260621 07:09:46.488731 24334 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:46.489125 24334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:46.489266 24334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:46.489384 24334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:46.497282 24334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:46.497385 24334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:46.497490 24334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:46.497586 24334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:46.497714 24334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:46.508354 24334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:43295,127.1.188.253:34113,127.1.188.252:43035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:43035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.188.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:46.509999 24334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:46.511673 24334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:46.525589 24339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:46.525439 24340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:46.525789 24342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:46.526762 24334 server_base.cc:1034] running on GCE node
I20260621 07:09:46.527786 24334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:46.529423 24334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:46.530702 24334 hybrid_clock.cc:648] HybridClock initialized: now 1782025786530634 us; error 32 us; skew 500 ppm
I20260621 07:09:46.531174 24334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:46.535168 24334 init.cc:377] Logged in from keytab as kudu/127.1.188.252@KRBTEST.COM (short username kudu)
I20260621 07:09:46.536968 24334 webserver.cc:460] Webserver started at http://127.1.188.252:41985/ using document root <none> and password file <none>
I20260621 07:09:46.537810 24334 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:46.537954 24334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:46.538331 24334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:46.541781 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data/instance:
uuid: "29ec4526da164686b668f4a377fcd65b"
format_stamp: "Formatted at 2026-06-21 07:09:46 on dist-test-slave-hlq1"
server_key: "e923174c9a8f48b8b5c02b528aec8f54"
server_key_iv: "2c141ce21122e904e184eeaae08cb27a"
server_key_version: "encryptionkey@0"
I20260621 07:09:46.542678 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal/instance:
uuid: "29ec4526da164686b668f4a377fcd65b"
format_stamp: "Formatted at 2026-06-21 07:09:46 on dist-test-slave-hlq1"
server_key: "e923174c9a8f48b8b5c02b528aec8f54"
server_key_iv: "2c141ce21122e904e184eeaae08cb27a"
server_key_version: "encryptionkey@0"
I20260621 07:09:46.548985 24334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260621 07:09:46.553189 24349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:46.554790 24334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 07:09:46.554988 24334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "29ec4526da164686b668f4a377fcd65b"
format_stamp: "Formatted at 2026-06-21 07:09:46 on dist-test-slave-hlq1"
server_key: "e923174c9a8f48b8b5c02b528aec8f54"
server_key_iv: "2c141ce21122e904e184eeaae08cb27a"
server_key_version: "encryptionkey@0"
I20260621 07:09:46.555238 24334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:46.588917 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 4
I20260621 07:09:46.590345 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.589822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55982)
0621 07:09:46.590062 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.590169 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 07:09:46.591640 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.591186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:55990)
0621 07:09:46.591360 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.591497 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 07:09:46.591843 24252 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:43035: Network error: Client connection negotiation failed: client connection to 127.1.188.252:43035: connect: Connection refused (error 111)
I20260621 07:09:46.592944 24334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:46.598001 24334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:46.598307 24334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:46.618280 24334 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.252:43035
I20260621 07:09:46.618301 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.252:43035 every 8 connection(s)
I20260621 07:09:46.619911 24334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
I20260621 07:09:46.624496 24401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:46.627288 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24334
I20260621 07:09:46.627542 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782025418728693-1779-0/minicluster-data/master-2/wal/instance
I20260621 07:09:46.628216 1779 external_mini_cluster.cc:1428] Setting key c3093d66b0a562929fea0178a0c6a57e
I20260621 07:09:46.633884 24401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:46.654102 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.637256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.1.188.254:43295)
0621 07:09:46.637816 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.637821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.638494 (+ 673us) server_negotiation.cc:407] Connection header received
0621 07:09:46.638775 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.638780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.638861 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.639004 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.640605 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.641372 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.642978 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.643533 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.643706 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.647490 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.647499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.647507 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.647564 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.651132 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.652007 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.652013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.652016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.652212 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.652640 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.652645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.652650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.652971 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.653126 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.653577 (+ 451us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.653765 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
I20260621 07:09:46.654750 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.637316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:43295 (local address 127.0.0.1:52440)
0621 07:09:46.638328 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.638373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.638580 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.639087 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.639095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.640036 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.640427 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.640438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.641537 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.641542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.642809 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.642821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.643782 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.643791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.643975 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.644673 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.644719 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.647226 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.651270 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.651278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.651285 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.651896 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.652321 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.652325 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.652328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.652507 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.653105 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.653114 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.653285 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.653488 (+ 203us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.653829 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":665,"threads_started":1}
I20260621 07:09:46.654794 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.630449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.1.188.252:43035)
0621 07:09:46.631466 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.631480 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.631532 (+ 52us) server_negotiation.cc:407] Connection header received
0621 07:09:46.633420 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.633445 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.633886 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.634254 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.636215 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.638382 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.640360 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.641015 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.641232 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.645625 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.645634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.645647 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.645678 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.649424 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.650286 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.650293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.650301 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.650455 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.650889 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.650893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.650895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.651707 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.652067 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.652569 (+ 502us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.653097 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"spinlock_wait_cycles":24448,"thread_start_us":461,"threads_started":1}
I20260621 07:09:46.656689 24401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:46.672086 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.657498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:34113 (local address 127.0.0.1:59436)
0621 07:09:46.657675 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.657695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.657814 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.658183 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.658188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.658723 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.658947 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.658962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.660248 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.660253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.661343 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.661355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.662095 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.662102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.662308 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.662903 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.662928 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.665310 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.669220 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.669224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.669227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.669748 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.670287 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.670291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.670293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.670483 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.671181 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.671184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.671354 (+ 170us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.671569 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.671809 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 07:09:46.672771 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.657586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.1.188.253:34113)
0621 07:09:46.657840 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.657848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.657869 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:09:46.657945 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.657953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.658038 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.658168 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.659139 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.660082 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.661494 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.661950 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.662049 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.665513 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.665518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.665522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.665566 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.669059 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.669919 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.669926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.669931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.670142 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.670612 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.670617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.670621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.671031 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.671223 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.672042 (+ 819us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.672298 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:09:46.673851 24401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:46.675478 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.658678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.1.188.254:43295)
0621 07:09:46.658840 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.658843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.658858 (+ 15us) server_negotiation.cc:407] Connection header received
0621 07:09:46.659056 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.659061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.659139 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.659243 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.660653 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.661366 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.662912 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.663317 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.663467 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.667889 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.667896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.667902 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.667944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.672309 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.673244 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.673252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.673257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.673396 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.673793 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.673800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.673803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.674166 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.674472 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.674798 (+ 326us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.675084 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025786, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:46.690040 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.674689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.1.188.252:43035)
0621 07:09:46.674870 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.674876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.675046 (+ 170us) server_negotiation.cc:407] Connection header received
0621 07:09:46.675199 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.675205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.675281 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.675460 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.676337 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.677424 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.679237 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.679923 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.680067 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.684213 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.684218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.684221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.684249 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.687239 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.687924 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.687928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.687930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.688061 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.688454 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.688458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.688459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.688871 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.688988 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.689429 (+ 441us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.689611 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:09:46.690326 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.674797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:56016)
0621 07:09:46.674950 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.674968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.675080 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.675630 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.675634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.676007 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.676214 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.676222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.677574 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.677579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.679088 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.679102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.680105 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.680110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.680399 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.681065 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.681089 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.684012 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.687367 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.687375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.687380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.687807 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.688172 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.688179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.688184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.688347 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.688990 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.688997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.689181 (+ 184us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.689351 (+ 170us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.689858 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":99}
I20260621 07:09:46.694320 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.677409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.1.188.253:34113)
0621 07:09:46.677571 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.677575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.677591 (+ 16us) server_negotiation.cc:407] Connection header received
0621 07:09:46.677680 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.677683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.677737 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.677812 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.679466 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.680500 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.682287 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.682863 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.683009 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.687847 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.687856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.687862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.687910 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.691501 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.692186 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.692193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.692197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.692401 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.692817 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.692823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.692827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.693119 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.693420 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.693712 (+ 292us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.693953 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 07:09:46.696696 24401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: Bootstrap starting.
I20260621 07:09:46.700239 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:46.701675 24401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:46.703048 24401 log.cc:824] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:46.703547 24227 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9fda74d732cc40a7982ae5ee4505729b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52446
I20260621 07:09:46.706671 24401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: No bootstrap required, opened a new log
I20260621 07:09:46.714927 24401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.715363 24401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ec4526da164686b668f4a377fcd65b, State: Initialized, Role: FOLLOWER
I20260621 07:09:46.715967 24401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.717088 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:46.717528 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:46.718374 24401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:09:46.730172 24419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:09:46.730300 24419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:09:46.727723 24334 master_runner.cc:186] Error getting master registration for 127.1.188.254:43295: OK, instance_id { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" instance_seqno: 1782025785441043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:09:46.737844 24334 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:09:46.743798 24326 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 5
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:09:46.760545 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.744710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:56024)
0621 07:09:46.744963 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.744985 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.745130 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.745544 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.745548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.746035 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.746307 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.746424 (+ 117us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.748270 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.748277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.749292 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.749305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.750234 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.750249 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.750491 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.751235 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.751262 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.753961 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.757641 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.757648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.757660 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.758090 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.758553 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.758560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.758564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.758739 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.759562 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.759566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.759703 (+ 137us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.759846 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.760202 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 07:09:46.760672 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.744850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.1.188.252:43035)
0621 07:09:46.745090 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.745096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.745119 (+ 23us) server_negotiation.cc:407] Connection header received
0621 07:09:46.745270 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.745279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.745390 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.745505 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.746639 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.748109 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.749452 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.750031 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.750203 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.754150 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.754155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.754159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.754194 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.757475 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.758282 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.758288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.758299 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.758432 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.758874 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.758879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.758883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.759427 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.759616 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.759977 (+ 361us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.760221 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:09:46.765578 24326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: Bootstrap starting.
I20260621 07:09:46.769173 24326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:46.770308 24326 log.cc:824] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:46.773113 24326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: No bootstrap required, opened a new log
I20260621 07:09:46.780591 24326 raft_consensus.cc:348] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.780931 24326 raft_consensus.cc:729] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b5c09070e40b18940414d5192e573, State: Initialized, Role: FOLLOWER
I20260621 07:09:46.781404 24326 consensus_queue.cc:260] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.782343 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:46.782794 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:46.783564 24326 sys_catalog.cc:564] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:09:46.793859 24433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:09:46.793982 24433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:09:46.791165 24259 master_runner.cc:186] Error getting master registration for 127.1.188.254:43295: OK, instance_id { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" instance_seqno: 1782025785441043 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:09:46.799487 24259 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 07:09:46.850805 24355 tablet.cc:2369] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:46.869062 24252 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } attempt: 5
Jun 21 07:09:46 dist-test-slave-hlq1 krb5kdc[24172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025785, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:09:46.885500 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.870192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.1.188.252:43035)
0621 07:09:46.870369 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.870377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.870399 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:09:46.870467 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.870471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.870550 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.870646 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:46.871892 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.872610 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.874436 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.874912 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.875052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.879326 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:46.879332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:46.879338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:46.879377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:46.882220 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.882981 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.882986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.882990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.883125 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:46.883676 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:46.883681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:46.883684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:46.884073 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:46.884216 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:46.884861 (+ 645us) server_negotiation.cc:299] Negotiation successful
0621 07:09:46.885128 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:09:46.885524 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.869900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:43035 (local address 127.0.0.1:56040)
0621 07:09:46.870105 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:46.870127 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:46.870277 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:46.870892 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:46.870897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:46.871494 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:46.871737 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.871750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.872758 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.872766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.874294 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:46.874309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:46.875063 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:46.875069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:46.875283 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:46.876029 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:46.876054 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:46.879134 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:46.882363 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.882370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.882375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.882865 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.883276 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:46.883280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:46.883284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:46.883538 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:46.884364 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:46.884370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:46.884544 (+ 174us) client_negotiation.cc:769] Sending connection context
0621 07:09:46.884689 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 07:09:46.885032 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 07:09:46.892066 24252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: Bootstrap starting.
I20260621 07:09:46.896121 24252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:46.897351 24252 log.cc:824] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:46.906258 24252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: No bootstrap required, opened a new log
I20260621 07:09:46.912858 24252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.913226 24252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fda74d732cc40a7982ae5ee4505729b, State: Initialized, Role: FOLLOWER
I20260621 07:09:46.913719 24252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.915081 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:46.915866 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:46.916291 24252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:46.916344 24227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:09:46.916532 24227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:46.921165 24227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.924032 24301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fda74d732cc40a7982ae5ee4505729b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a27b5c09070e40b18940414d5192e573"
I20260621 07:09:46.924619 24301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:46.929343 24301 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fda74d732cc40a7982ae5ee4505729b in term 1.
I20260621 07:09:46.930248 24201 leader_election.cc:302] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fda74d732cc40a7982ae5ee4505729b, a27b5c09070e40b18940414d5192e573; no voters:
W20260621 07:09:46.930646 24206 tablet.cc:2369] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:46.932011 24376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fda74d732cc40a7982ae5ee4505729b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29ec4526da164686b668f4a377fcd65b"
I20260621 07:09:46.932891 24376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:46.938323 24376 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fda74d732cc40a7982ae5ee4505729b in term 1.
I20260621 07:09:46.938623 24227 leader_election.cc:288] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [CANDIDATE]: Term 1 election: Requested vote from peers a27b5c09070e40b18940414d5192e573 (127.1.188.253:34113), 29ec4526da164686b668f4a377fcd65b (127.1.188.252:43035)
I20260621 07:09:46.946457 24436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:46.955665 24436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [term 1 LEADER]: Becoming Leader. State: Replica: 9fda74d732cc40a7982ae5ee4505729b, State: Running, Role: LEADER
I20260621 07:09:46.957177 24436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } }
I20260621 07:09:46.968195 24442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fda74d732cc40a7982ae5ee4505729b. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:46.968648 24442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:46.984522 24450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260621 07:09:46.996752 24280 tablet.cc:2369] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:47.006695 24450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:47.032322 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.984308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.1.188.252:43035)
0621 07:09:46.984554 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.984563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.997312 (+ 12749us) server_negotiation.cc:407] Connection header received
0621 07:09:46.997870 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.997876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.997973 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.998047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.003516 (+ 5469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.004674 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.010417 (+ 5743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.011436 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.011564 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:47.018743 (+ 7179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:47.018750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:47.018754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:47.018797 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:47.022282 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.027994 (+ 5712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.028003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.028009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.028178 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.029241 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.029246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.029249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.029758 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:47.030086 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:47.031618 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0621 07:09:47.031889 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260621 07:09:47.037544 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.994971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.1.188.254:43295)
0621 07:09:46.995167 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.995176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.998864 (+ 3688us) server_negotiation.cc:407] Connection header received
0621 07:09:47.000935 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.000941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.001029 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.001113 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.005604 (+ 4491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.006761 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.015357 (+ 8596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.016048 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.016157 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:47.024259 (+ 8102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:47.024267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:47.024272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:47.024313 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:47.027803 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.030913 (+ 3110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.030922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.030928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.031109 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.035328 (+ 4219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.035337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.035344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.035800 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:47.036083 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:47.036792 (+ 709us) server_negotiation.cc:299] Negotiation successful
0621 07:09:47.037117 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 07:09:47.040954 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:46.977978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.1.188.253:34113)
0621 07:09:46.978319 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:46.978329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:46.997095 (+ 18766us) server_negotiation.cc:407] Connection header received
0621 07:09:46.997558 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:46.997565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:46.997658 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:46.997755 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.001822 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.003056 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.006212 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.006905 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.007000 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:47.031372 (+ 24372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:47.031380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:47.031384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:47.031450 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:47.035641 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.037115 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.037120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.037123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.037271 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:47.038038 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:47.038043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:47.038046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:47.038454 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:47.038689 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:47.040177 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0621 07:09:47.040444 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 07:09:47.041232 24376 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [term 1 FOLLOWER]: Refusing update from remote peer 9fda74d732cc40a7982ae5ee4505729b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:47.042325 24442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:47.044745 24301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [term 1 FOLLOWER]: Refusing update from remote peer 9fda74d732cc40a7982ae5ee4505729b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:47.044906 24458 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 07:09:47.046103 24442 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:47.060956 24459 mvcc.cc:205] Tried to move back new op lower bound from 7299177623641567232 to 7299177623393619968. Current Snapshot: MvccSnapshot[applied={T|T < 7299177623641567232}]
I20260621 07:09:47.061164 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fda74d732cc40a7982ae5ee4505729b. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.061523 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:47.063824 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.064123 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:47.064687 24461 mvcc.cc:205] Tried to move back new op lower bound from 7299177623641567232 to 7299177623393619968. Current Snapshot: MvccSnapshot[applied={T|T < 7299177623641567232}]
I20260621 07:09:47.065829 24450 catalog_manager.cc:1347] Generated new cluster ID: a23222b2328543ff8403a41dd0f012b7
I20260621 07:09:47.065922 24450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:47.086323 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fda74d732cc40a7982ae5ee4505729b. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.086771 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:47.096066 24442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.096450 24442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:47.099567 24408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.099886 24408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:47.105798 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fda74d732cc40a7982ae5ee4505729b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fda74d732cc40a7982ae5ee4505729b" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 43295 } } peers { permanent_uuid: "a27b5c09070e40b18940414d5192e573" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 34113 } } peers { permanent_uuid: "29ec4526da164686b668f4a377fcd65b" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 43035 } } }
I20260621 07:09:47.106040 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:47.122388 24450 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:47.124251 24450 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:47.140079 24450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fda74d732cc40a7982ae5ee4505729b: Generated new TSK 0
I20260621 07:09:47.147522 24450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 07:09:47.280066 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.264517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.1.188.253:34113)
0621 07:09:47.264740 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.264747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.278054 (+ 13307us) server_negotiation.cc:407] Connection header received
0621 07:09:47.278370 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.278378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.278471 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.278562 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.279800 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59464: BlockingRecv error: recv got EOF from 127.0.0.1:59464 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 07:09:47.282502 24404 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.270634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.1.188.252:43035)
0621 07:09:47.270834 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.270841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.280838 (+ 9997us) server_negotiation.cc:407] Connection header received
0621 07:09:47.281097 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.281103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.281189 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.281268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.282262 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56058: BlockingRecv error: recv got EOF from 127.0.0.1:56058 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 07:09:47.284606 24405 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.276687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.1.188.254:43295)
0621 07:09:47.276926 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.276933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.282531 (+ 5598us) server_negotiation.cc:407] Connection header received
0621 07:09:47.282719 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.282725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.282807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.282910 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.284358 (+ 1448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52466: BlockingRecv error: recv got EOF from 127.0.0.1:52466 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260621 07:09:47.330447 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.317752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.1.188.253:34113)
0621 07:09:47.318003 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.318013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.328536 (+ 10523us) server_negotiation.cc:407] Connection header received
0621 07:09:47.328765 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.328772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.328929 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.329050 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.330114 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59466: BlockingRecv error: recv got EOF from 127.0.0.1:59466 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 07:09:47.397940 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.379797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.1.188.253:34113)
0621 07:09:47.379985 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.379993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.396048 (+ 16055us) server_negotiation.cc:407] Connection header received
0621 07:09:47.396311 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.396317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.396426 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.396515 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.397672 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 07:09:47.431226 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.426158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.1.188.253:34113)
0621 07:09:47.426340 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.426347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.429701 (+ 3354us) server_negotiation.cc:407] Connection header received
0621 07:09:47.429889 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.429895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.430002 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.430119 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.430955 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59474: BlockingRecv error: recv got EOF from 127.0.0.1:59474 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260621 07:09:47.440918 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.438892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.1.188.253:34113)
0621 07:09:47.439121 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.439127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.439430 (+ 303us) server_negotiation.cc:407] Connection header received
0621 07:09:47.439648 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.439653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.439733 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.439844 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.440612 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59490: BlockingRecv error: recv got EOF from 127.0.0.1:59490 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260621 07:09:47.454041 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.452050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.1.188.253:34113)
0621 07:09:47.452215 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.452222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.452599 (+ 377us) server_negotiation.cc:407] Connection header received
0621 07:09:47.452786 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.452791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.452892 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.453010 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.453815 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59504: BlockingRecv error: recv got EOF from 127.0.0.1:59504 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260621 07:09:47.475504 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.473522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.1.188.253:34113)
0621 07:09:47.473722 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.473729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.474062 (+ 333us) server_negotiation.cc:407] Connection header received
0621 07:09:47.474236 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.474242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.474349 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.474481 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.475202 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59510: BlockingRecv error: recv got EOF from 127.0.0.1:59510 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 07:09:47.515095 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.512861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.1.188.253:34113)
0621 07:09:47.513077 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.513084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.513425 (+ 341us) server_negotiation.cc:407] Connection header received
0621 07:09:47.513655 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.513659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.513752 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.513834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.514817 (+ 983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59514: BlockingRecv error: recv got EOF from 127.0.0.1:59514 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260621 07:09:47.585152 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.583122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.1.188.253:34113)
0621 07:09:47.583346 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.583353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.583793 (+ 440us) server_negotiation.cc:407] Connection header received
0621 07:09:47.583970 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.583975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.584066 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.584185 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.584927 (+ 742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59520: BlockingRecv error: recv got EOF from 127.0.0.1:59520 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 07:09:47.720961 24331 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:47.718863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.1.188.253:34113)
0621 07:09:47.719121 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.719128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.719497 (+ 369us) server_negotiation.cc:407] Connection header received
0621 07:09:47.719689 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.719693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.719769 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.719858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:47.720695 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59526: BlockingRecv error: recv got EOF from 127.0.0.1:59526 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 07:09:47.732486 24419 catalog_manager.cc:1259] Loaded cluster ID: a23222b2328543ff8403a41dd0f012b7
I20260621 07:09:47.732584 24419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: loading cluster ID for follower catalog manager: success
I20260621 07:09:47.735329 24419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: acquiring CA information for follower catalog manager: success
I20260621 07:09:47.737248 24419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29ec4526da164686b668f4a377fcd65b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:09:47.796553 24433 catalog_manager.cc:1259] Loaded cluster ID: a23222b2328543ff8403a41dd0f012b7
I20260621 07:09:47.796677 24433 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: loading cluster ID for follower catalog manager: success
I20260621 07:09:47.799624 24433 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: acquiring CA information for follower catalog manager: success
I20260621 07:09:47.801620 24433 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a27b5c09070e40b18940414d5192e573: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:09:47.992049 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:47.985590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.1.188.253:34113)
0621 07:09:47.985878 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:47.985886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:47.986149 (+ 263us) server_negotiation.cc:407] Connection header received
0621 07:09:47.986397 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:47.986402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:47.986598 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:47.986720 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:47.987306 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.988078 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.989441 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:47.989886 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:47.989987 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:47.990625 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:47.991327 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:47.991527 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:47.991574 (+ 47us) server_negotiation.cc:299] Negotiation successful
0621 07:09:47.991658 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260621 07:09:48.006103 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:47.999682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.1.188.252:43035)
0621 07:09:48.000206 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:48.000211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:48.000279 (+ 68us) server_negotiation.cc:407] Connection header received
0621 07:09:48.000498 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:48.000502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:48.000699 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:48.000862 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:48.001313 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.002145 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.003619 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.004030 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.004131 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:48.004718 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:48.005419 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:48.005569 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:48.005669 (+ 100us) server_negotiation.cc:299] Negotiation successful
0621 07:09:48.005759 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":281,"threads_started":1}
I20260621 07:09:48.025012 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:48.013877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.1.188.254:43295)
0621 07:09:48.016021 (+ 2144us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:48.016030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:48.016767 (+ 737us) server_negotiation.cc:407] Connection header received
0621 07:09:48.016925 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:48.016934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:48.017230 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:48.018058 (+ 828us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:48.018248 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.019518 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.021808 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.022485 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.023100 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:48.023337 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:48.023939 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:48.024496 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:48.024550 (+ 54us) server_negotiation.cc:299] Negotiation successful
0621 07:09:48.024652 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1782,"threads_started":1}
I20260621 07:09:48.026280 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:48.014200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.1.188.252:43035)
0621 07:09:48.015942 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:48.015949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:48.018417 (+ 2468us) server_negotiation.cc:407] Connection header received
0621 07:09:48.019077 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:48.019083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:48.019255 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:48.019343 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:48.020456 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.021637 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.023853 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.024545 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.024642 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:48.025416 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:48.025565 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:48.025648 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:48.025846 (+ 198us) server_negotiation.cc:299] Negotiation successful
0621 07:09:48.025937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589}
I20260621 07:09:48.028402 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:48.013562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.1.188.253:34113)
0621 07:09:48.013854 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:48.013865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:48.020314 (+ 6449us) server_negotiation.cc:407] Connection header received
0621 07:09:48.020801 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:48.020808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:48.020920 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:48.020995 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:48.021815 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.023017 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.025704 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:48.026419 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:48.026541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:48.027508 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:48.027656 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:48.027748 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:48.027945 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 07:09:48.028056 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 07:09:48.032800 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24185
I20260621 07:09:48.033990 24247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:48.159919 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24185
W20260621 07:09:48.184475 24275 connection.cc:535] server connection from 127.0.0.1:59420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:09:48.185019 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24259
I20260621 07:09:48.186370 24321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:48.332082 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24259
I20260621 07:09:48.356122 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24334
I20260621 07:09:48.357416 24396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:48.480588 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24334
2026-06-21T07:09:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4901 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 07:09:48 dist-test-slave-hlq1 krb5kdc[24546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 07:09:48 dist-test-slave-hlq1 krb5kdc[24546](info): set up 2 sockets
Jun 21 07:09:48 dist-test-slave-hlq1 krb5kdc[24546](info): commencing operation
krb5kdc: starting...
W20260621 07:09:49.629524 1779 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/test-user.keytab.
Jun 21 07:09:49 dist-test-slave-hlq1 krb5kdc[24546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T07:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T07:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.188.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
Entry for principal kudu/127.1.188.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab.
I20260621 07:09:49.925036 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41741
--webserver_interface=127.1.188.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 07:09:50.275059 24559 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:50.275369 24559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:50.275508 24559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:50.275597 24559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:50.282657 24559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:50.282739 24559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:50.282796 24559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:50.282840 24559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:50.282900 24559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:50.292809 24559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.254.keytab
--principal=kudu/127.1.188.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.254:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.188.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:50.294108 24559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:50.295372 24559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:50.307186 24564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:50.307150 24565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:50.307237 24567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:50.308082 24559 server_base.cc:1034] running on GCE node
I20260621 07:09:50.308872 24559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:50.310256 24559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:50.311555 24559 hybrid_clock.cc:648] HybridClock initialized: now 1782025790311489 us; error 36 us; skew 500 ppm
I20260621 07:09:50.311923 24559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:50 dist-test-slave-hlq1 krb5kdc[24546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:50.315070 24559 init.cc:377] Logged in from keytab as kudu/127.1.188.254@KRBTEST.COM (short username kudu)
I20260621 07:09:50.316648 24559 webserver.cc:460] Webserver started at http://127.1.188.254:39873/ using document root <none> and password file <none>
I20260621 07:09:50.317360 24559 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:50.317494 24559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:50.317843 24559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:50.320837 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data/instance:
uuid: "c4ab2e4ddec246e494da465afe38958a"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "6f6b8e16079eaf7e0d144986dc9af845"
server_key_iv: "e4a817d1ece23231e3bbeca188b090f3"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.321627 24559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance:
uuid: "c4ab2e4ddec246e494da465afe38958a"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "6f6b8e16079eaf7e0d144986dc9af845"
server_key_iv: "e4a817d1ece23231e3bbeca188b090f3"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.326691 24559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 07:09:50.330195 24574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:50.331591 24559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 07:09:50.331786 24559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal
uuid: "c4ab2e4ddec246e494da465afe38958a"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "6f6b8e16079eaf7e0d144986dc9af845"
server_key_iv: "e4a817d1ece23231e3bbeca188b090f3"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.332010 24559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:50.356276 24559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:50.364517 24559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:50.364818 24559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:50.384110 24559 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.254:41741
I20260621 07:09:50.384133 24625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.254:41741 every 8 connection(s)
I20260621 07:09:50.385464 24559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/data/info.pb
I20260621 07:09:50.389178 24626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:50.391309 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24559
I20260621 07:09:50.391561 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-0/wal/instance
I20260621 07:09:50.392185 1779 external_mini_cluster.cc:1428] Setting key 4541a43c2db48554273e63acf6b0d26f
I20260621 07:09:50.396484 24626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:50 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.253@KRBTEST.COM" created.
I20260621 07:09:50.423357 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.399790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.1.188.254:41741)
0621 07:09:50.400647 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:50.400663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:50.401209 (+ 546us) server_negotiation.cc:407] Connection header received
0621 07:09:50.403598 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:50.403609 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:50.403938 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:50.404167 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:50.405448 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.407635 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.409077 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.410137 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.410360 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.414286 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:50.414300 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:50.414320 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:50.414372 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:50.418767 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.419582 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.419595 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.419603 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.419819 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.420326 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.420332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.420336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.420937 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:50.421149 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:50.421832 (+ 683us) server_negotiation.cc:299] Negotiation successful
0621 07:09:50.422081 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":394,"threads_started":1}
I20260621 07:09:50.423449 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.399802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41741 (local address 127.0.0.1:47662)
0621 07:09:50.401010 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.401055 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:50.403163 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:50.404217 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:50.404223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:50.404862 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:50.405212 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.405229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.407815 (+ 2586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.407826 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.408927 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.408939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.410415 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.410421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.410554 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.411375 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:50.411457 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:50.414018 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:50.418938 (+ 4920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.418947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.418957 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.419347 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.419950 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.419953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.419956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.420113 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.421124 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:50.421132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:50.421415 (+ 283us) client_negotiation.cc:769] Sending connection context
0621 07:09:50.421631 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 07:09:50.422239 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":67,"spinlock_wait_cycles":16640,"thread_start_us":715,"threads_started":1}
I20260621 07:09:50.426626 24626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:50.427964 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.427467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58708)
0621 07:09:50.427669 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.427815 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 07:09:50.428264 24577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
I20260621 07:09:50.430058 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.429567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58718)
0621 07:09:50.429776 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.429876 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260621 07:09:50.430301 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:43877: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
Entry for principal kudu/127.1.188.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab.
I20260621 07:09:50.454234 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:43877
--webserver_interface=127.1.188.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:09:50.479660 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } attempt: 1
I20260621 07:09:50.481015 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.480447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58734)
0621 07:09:50.480669 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.480852 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 07:09:50.482164 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.481779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58748)
0621 07:09:50.481926 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.482010 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:09:50.482424 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:43877: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
I20260621 07:09:50.550741 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } attempt: 2
I20260621 07:09:50.552196 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.551572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58754)
0621 07:09:50.551894 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.552019 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 07:09:50.553472 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.553008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58756)
0621 07:09:50.553193 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.553280 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260621 07:09:50.553764 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:43877: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
I20260621 07:09:50.645028 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } attempt: 3
I20260621 07:09:50.646373 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.645744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58762)
0621 07:09:50.646022 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.646159 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260621 07:09:50.647710 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.647166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58772)
0621 07:09:50.647354 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.647499 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 07:09:50.647979 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:43877: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
W20260621 07:09:50.788141 24633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:50.788513 24633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:50.788635 24633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:50.788785 24633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260621 07:09:50.791299 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } attempt: 4
I20260621 07:09:50.792594 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.791987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58776)
0621 07:09:50.792306 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.792415 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260621 07:09:50.793707 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.793297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58784)
0621 07:09:50.793457 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.793555 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:09:50.793910 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.253:43877: Network error: Client connection negotiation failed: client connection to 127.1.188.253:43877: connect: Connection refused (error 111)
W20260621 07:09:50.796859 24633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:50.796937 24633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:50.797051 24633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:50.797114 24633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:50.797212 24633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:50.807183 24633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.253.keytab
--principal=kudu/127.1.188.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.253:43877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.188.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:50.808481 24633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:50.809793 24633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 07:09:50.821345 24639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:50.821380 24641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:50.821300 24638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:50.822486 24633 server_base.cc:1034] running on GCE node
I20260621 07:09:50.823202 24633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:50.824582 24633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:50.825819 24633 hybrid_clock.cc:648] HybridClock initialized: now 1782025790825743 us; error 43 us; skew 500 ppm
I20260621 07:09:50.826100 24633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 07:09:50 dist-test-slave-hlq1 krb5kdc[24546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:50.829447 24633 init.cc:377] Logged in from keytab as kudu/127.1.188.253@KRBTEST.COM (short username kudu)
I20260621 07:09:50.830909 24633 webserver.cc:460] Webserver started at http://127.1.188.253:40391/ using document root <none> and password file <none>
I20260621 07:09:50.831681 24633 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:50.831825 24633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:50.832147 24633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:50.835095 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data/instance:
uuid: "006a6dc75e60484bade735b23468d12f"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "d456e26dfad5c96a4b1d8cc14d9fcba2"
server_key_iv: "d398fe9e26bcb78be6215271ef3e1a07"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.835932 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal/instance:
uuid: "006a6dc75e60484bade735b23468d12f"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "d456e26dfad5c96a4b1d8cc14d9fcba2"
server_key_iv: "d398fe9e26bcb78be6215271ef3e1a07"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.841033 24633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:09:50.844491 24648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:50.845719 24633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 07:09:50.845899 24633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal
uuid: "006a6dc75e60484bade735b23468d12f"
format_stamp: "Formatted at 2026-06-21 07:09:50 on dist-test-slave-hlq1"
server_key: "d456e26dfad5c96a4b1d8cc14d9fcba2"
server_key_iv: "d398fe9e26bcb78be6215271ef3e1a07"
server_key_version: "encryptionkey@0"
I20260621 07:09:50.846112 24633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:50.881472 24633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:50.885972 24633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:50.886300 24633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:50.906352 24633 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.253:43877
I20260621 07:09:50.906374 24699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.253:43877 every 8 connection(s)
I20260621 07:09:50.907855 24633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/data/info.pb
I20260621 07:09:50.909762 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24633
I20260621 07:09:50.909971 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-1/wal/instance
I20260621 07:09:50.910458 1779 external_mini_cluster.cc:1428] Setting key fe7cc847d0ffe3406137a6eb67b5e188
I20260621 07:09:50.911866 24700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:50.918892 24700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:50 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.188.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.188.252@KRBTEST.COM" created.
I20260621 07:09:50.941968 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.922621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.1.188.254:41741)
0621 07:09:50.922867 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:50.922876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:50.924292 (+ 1416us) server_negotiation.cc:407] Connection header received
0621 07:09:50.926197 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:50.926214 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:50.926320 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:50.926448 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:50.927988 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.929181 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.930728 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.931573 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.931733 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.935528 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:50.935535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:50.935549 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:50.935586 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:50.938681 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.939607 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.939612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.939614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.939747 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.940183 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.940187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.940189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.940482 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:50.940643 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:50.941379 (+ 736us) server_negotiation.cc:299] Negotiation successful
0621 07:09:50.941633 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 07:09:50.942979 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.923167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41741 (local address 127.0.0.1:47666)
0621 07:09:50.924107 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.924157 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:50.925926 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:50.926631 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:50.926639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:50.927188 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:50.927808 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.927825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.929360 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.929364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.930572 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.930584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.931809 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.931815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.932022 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.932798 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:50.932821 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:50.935225 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:50.938855 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.938864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.938888 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.939466 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.939863 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.939868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.939872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.940056 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.940619 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:50.940628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:50.940991 (+ 363us) client_negotiation.cc:769] Sending connection context
0621 07:09:50.941267 (+ 276us) client_negotiation.cc:240] Negotiation successful
0621 07:09:50.941817 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":449,"threads_started":1}
I20260621 07:09:50.945055 24700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:50 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:50.961155 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.945999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58794)
0621 07:09:50.946167 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.946185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:50.946312 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:50.947294 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:50.947302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:50.947866 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:50.948081 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.948096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.949642 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.949650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.951280 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:50.951296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.952165 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.952170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:50.952408 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.953001 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:50.953018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:50.955228 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:50.958545 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.958552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.958556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.958925 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.959311 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:50.959316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:50.959319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:50.959554 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:50.960146 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:50.960151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:50.960285 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 07:09:50.960435 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 07:09:50.960713 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":26}
I20260621 07:09:50.961191 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.946057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.1.188.253:43877)
0621 07:09:50.946883 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:50.946889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:50.946911 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:09:50.946950 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:50.946956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:50.947117 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:50.947237 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:50.948315 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.949511 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.951482 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:50.952021 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:50.952157 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:50.955466 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:50.955479 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:50.955499 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:50.955526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:50.958405 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.959059 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.959068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.959071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.959189 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:50.959657 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:50.959664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:50.959667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:50.960010 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:50.960225 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:50.960566 (+ 341us) server_negotiation.cc:299] Negotiation successful
0621 07:09:50.960817 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":681,"thread_start_us":538,"threads_started":1}
I20260621 07:09:50.963508 24700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:50.964949 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.964355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36346)
0621 07:09:50.964569 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.964706 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260621 07:09:50.965229 24651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:50.966987 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:50.966566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36356)
0621 07:09:50.966714 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:50.966802 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260621 07:09:50.967239 24700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
Entry for principal kudu/127.1.188.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab.
I20260621 07:09:50.970708 1779 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRiL99G/build/asan/bin/kudu
/tmp/dist-test-taskRiL99G/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:42387
--webserver_interface=127.1.188.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.188.212:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 07:09:51.016597 24700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 1
I20260621 07:09:51.017807 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.017343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36370)
0621 07:09:51.017533 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.017651 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 07:09:51.018967 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.018578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36382)
0621 07:09:51.018759 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.018836 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 07:09:51.019168 24700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.087456 24700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 2
I20260621 07:09:51.088652 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.088160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36392)
0621 07:09:51.088410 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.088499 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260621 07:09:51.089730 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.089374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36404)
0621 07:09:51.089554 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.089619 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260621 07:09:51.089960 24700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.093142 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } attempt: 5
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:51.107932 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.093792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58798)
0621 07:09:51.093979 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.094003 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.094112 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.094485 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.094489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.094878 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.095064 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.095073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.096471 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.096475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.097378 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.097390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.098526 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.098530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.098700 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.099350 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.099369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.101477 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.105344 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.105349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.105352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.105865 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.106298 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.106301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.106304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.106428 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.107092 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.107096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.107256 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.107382 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.107674 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 07:09:51.108067 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.093904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.1.188.253:43877)
0621 07:09:51.094081 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.094088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.094110 (+ 22us) server_negotiation.cc:407] Connection header received
0621 07:09:51.094236 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.094241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.094337 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.094439 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.095220 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.096328 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.097527 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.098368 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.098506 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.101674 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.101681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.101686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.101734 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.105140 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.105989 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.105997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.106001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.106149 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.106530 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.106537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.106543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.106939 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.107120 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.107529 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.107776 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 07:09:51.109239 24626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
I20260621 07:09:51.110316 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.109907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36414)
0621 07:09:51.110063 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.110137 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 07:09:51.111194 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.110886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36424)
0621 07:09:51.111044 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.111104 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 07:09:51.111406 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.162672 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 1
I20260621 07:09:51.164016 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.163513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36432)
0621 07:09:51.163718 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.163845 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:09:51.165002 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.164669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36448)
0621 07:09:51.164831 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.164897 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 07:09:51.165215 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.181237 24700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 3
I20260621 07:09:51.182423 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.181951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36462)
0621 07:09:51.182158 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.182257 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:09:51.183670 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.183144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36468)
0621 07:09:51.183438 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.183541 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260621 07:09:51.183884 24700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.233460 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 2
I20260621 07:09:51.234648 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.234155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36476)
0621 07:09:51.234396 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.234492 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260621 07:09:51.235792 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.235455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36492)
0621 07:09:51.235590 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.235670 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260621 07:09:51.236021 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
W20260621 07:09:51.303268 24707 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 07:09:51.303732 24707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 07:09:51.303859 24707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 07:09:51.303998 24707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 07:09:51.311489 24707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 07:09:51.311582 24707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 07:09:51.311683 24707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 07:09:51.311765 24707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 07:09:51.311859 24707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 07:09:51.321828 24707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.188.212:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.188.254:41741,127.1.188.253:43877,127.1.188.252:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/krb5kdc/kudu_127.1.188.252.keytab
--principal=kudu/127.1.188.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.188.252:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.188.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 05:57:27 UTC on ccycloud.quasar-zbapqx.root.comops.site
build id 4457935
ASAN enabled
I20260621 07:09:51.323137 24707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 07:09:51.324543 24707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 07:09:51.327111 24700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 4
I20260621 07:09:51.328406 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.327832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36498)
0621 07:09:51.328091 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.328204 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 07:09:51.329599 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.329158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36506)
0621 07:09:51.329324 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.329428 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 07:09:51.329849 24700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
W20260621 07:09:51.336449 24712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:51.336405 24713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 07:09:51.336480 24715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 07:09:51.337605 24707 server_base.cc:1034] running on GCE node
I20260621 07:09:51.338409 24707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 07:09:51.339812 24707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 07:09:51.341048 24707 hybrid_clock.cc:648] HybridClock initialized: now 1782025791340965 us; error 55 us; skew 500 ppm
I20260621 07:09:51.341336 24707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 07:09:51.342291 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 3
I20260621 07:09:51.343292 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.342837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36510)
0621 07:09:51.343059 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.343134 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025791, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 07:09:51.344480 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.344045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36522)
0621 07:09:51.344259 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.344347 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260621 07:09:51.344739 24626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.188.252:42387: Network error: Client connection negotiation failed: client connection to 127.1.188.252:42387: connect: Connection refused (error 111)
I20260621 07:09:51.345064 24707 init.cc:377] Logged in from keytab as kudu/127.1.188.252@KRBTEST.COM (short username kudu)
I20260621 07:09:51.346585 24707 webserver.cc:460] Webserver started at http://127.1.188.252:33091/ using document root <none> and password file <none>
I20260621 07:09:51.347231 24707 fs_manager.cc:359] Metadata directory not provided
I20260621 07:09:51.347318 24707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 07:09:51.347651 24707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 07:09:51.350646 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data/instance:
uuid: "d263a4fd08e04fd0b4af494dcdf0921f"
format_stamp: "Formatted at 2026-06-21 07:09:51 on dist-test-slave-hlq1"
server_key: "3d639bbaeb08a70b1faea10a4420f6ca"
server_key_iv: "1aa5d7b468ea90c66b6472486fcd2698"
server_key_version: "encryptionkey@0"
I20260621 07:09:51.351509 24707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal/instance:
uuid: "d263a4fd08e04fd0b4af494dcdf0921f"
format_stamp: "Formatted at 2026-06-21 07:09:51 on dist-test-slave-hlq1"
server_key: "3d639bbaeb08a70b1faea10a4420f6ca"
server_key_iv: "1aa5d7b468ea90c66b6472486fcd2698"
server_key_version: "encryptionkey@0"
I20260621 07:09:51.356712 24707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 07:09:51.360150 24722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 07:09:51.361517 24707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 07:09:51.361703 24707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data,/tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal
uuid: "d263a4fd08e04fd0b4af494dcdf0921f"
format_stamp: "Formatted at 2026-06-21 07:09:51 on dist-test-slave-hlq1"
server_key: "3d639bbaeb08a70b1faea10a4420f6ca"
server_key_iv: "1aa5d7b468ea90c66b6472486fcd2698"
server_key_version: "encryptionkey@0"
I20260621 07:09:51.361928 24707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 07:09:51.376485 24707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 07:09:51.380931 24707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 07:09:51.381175 24707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 07:09:51.401291 24707 rpc_server.cc:307] RPC server started. Bound to: 127.1.188.252:42387
I20260621 07:09:51.401345 24773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.188.252:42387 every 8 connection(s)
I20260621 07:09:51.402791 24707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/data/info.pb
I20260621 07:09:51.405467 1779 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRiL99G/build/asan/bin/kudu as pid 24707
I20260621 07:09:51.405673 1779 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRiL99G/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782025418728693-1779-0/minicluster-data/master-2/wal/instance
I20260621 07:09:51.406149 1779 external_mini_cluster.cc:1428] Setting key 1749b190c1228d2135848b206e0adce0
I20260621 07:09:51.406635 24774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 07:09:51.415459 24774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025791, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
I20260621 07:09:51.430217 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.408066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.1.188.252:42387)
0621 07:09:51.408815 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.408827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.408866 (+ 39us) server_negotiation.cc:407] Connection header received
0621 07:09:51.411075 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.411096 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.411634 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.412039 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.413531 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.414788 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.416155 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.416897 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.417090 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.421578 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.421589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.421602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.421657 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.425803 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.426478 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.426484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.426490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.426632 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.427067 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.427071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.427073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.427718 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.428048 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.428413 (+ 365us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.428857 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":347,"threads_started":1}
I20260621 07:09:51.434334 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.418078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.1.188.254:41741)
0621 07:09:51.418351 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.418360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.418826 (+ 466us) server_negotiation.cc:407] Connection header received
0621 07:09:51.419031 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.419037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.419130 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.419258 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.420408 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.421149 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.423252 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.423751 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.423849 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.427637 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.427644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.427650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.427691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.431380 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.432301 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.432305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.432307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.432431 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.432822 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.432828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.432831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.433131 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.433364 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.433888 (+ 524us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.434046 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260621 07:09:51.435170 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.417935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.254:41741 (local address 127.0.0.1:47674)
0621 07:09:51.418663 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.418714 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.418878 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.419362 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.419366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.419994 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.420241 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.420250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.421351 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.421357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.423093 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.423113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.423938 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.423945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.424155 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.424724 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.424747 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.427384 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.431546 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.431555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.431562 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.432169 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.432543 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.432551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.432555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.432696 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.433310 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.433320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.433502 (+ 182us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.434594 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.434856 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":343,"threads_started":1}
I20260621 07:09:51.436887 24774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.254@KRBTEST.COM
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025791, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:51.450480 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.433425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.1.188.254:41741)
0621 07:09:51.434002 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.434007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.434025 (+ 18us) server_negotiation.cc:407] Connection header received
0621 07:09:51.434062 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.434065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.434117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.434253 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.435486 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.436643 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.438263 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.438898 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.439027 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.443574 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.443583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.443588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.443628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.447210 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.448005 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.448014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.448017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.448153 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.448566 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.448572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.448576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.448979 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.449229 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.449830 (+ 601us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.450056 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":276,"threads_started":1}
I20260621 07:09:51.452018 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.437666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.253:43877 (local address 127.0.0.1:58804)
0621 07:09:51.437824 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.437841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.437924 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.438271 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.438275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.438696 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.438884 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.438894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.440298 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.440302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.441193 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.441206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.442173 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.442180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.442462 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.443042 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.443058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.445409 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.449116 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.449119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.449122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.449757 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.450164 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.450169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.450173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.450348 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.451027 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.451032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.451170 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.451342 (+ 172us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.451649 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 07:09:51.452072 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.437694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.1.188.253:43877)
0621 07:09:51.437866 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.437873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.437893 (+ 20us) server_negotiation.cc:407] Connection header received
0621 07:09:51.438040 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.438046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.438126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.438231 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.439038 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.440125 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.441339 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.442009 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.442101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.445616 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.445622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.445627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.445665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.448988 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.449869 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.449875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.449878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.450032 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.450494 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.450517 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.450523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.450851 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.451008 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.451473 (+ 465us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.451678 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 07:09:51.453306 24774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025791, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.252@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.188.253@KRBTEST.COM
I20260621 07:09:51.466266 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.454168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.1.188.252:42387)
0621 07:09:51.454345 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.454351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.454365 (+ 14us) server_negotiation.cc:407] Connection header received
0621 07:09:51.454408 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.454411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.454457 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.454553 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.455178 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.455931 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.457012 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.457420 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.457560 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.460635 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.460642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.460649 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.460689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.463657 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.464287 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.464295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.464298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.464429 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.464773 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.464781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.464785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.465121 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.465294 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.465631 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.465869 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 07:09:51.466514 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.453941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36538)
0621 07:09:51.454104 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.454155 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.454252 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.454567 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.454570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.454935 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.455052 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.455060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.456049 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.456057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.456882 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.456892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.457574 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.457580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.457783 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.458279 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.458294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.460461 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.463802 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.463809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.463813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.464174 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.464533 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.464537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.464543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.464666 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.465266 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.465271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.465414 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.465559 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.466017 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":148}
I20260621 07:09:51.467037 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.452149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.1.188.253:43877)
0621 07:09:51.452378 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.452386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.452767 (+ 381us) server_negotiation.cc:407] Connection header received
0621 07:09:51.452974 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.452979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.453049 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.453149 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.454501 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.455211 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.456467 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.456882 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.457014 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.461668 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.461674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.461677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.461708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.464354 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.465063 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.465069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.465073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.465205 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.465625 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.465631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.465634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.465980 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.466266 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.466587 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.466801 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":628,"threads_started":1}
I20260621 07:09:51.472043 24774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: Bootstrap starting.
I20260621 07:09:51.472285 1779 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 07:09:51.475198 24774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:51.476343 24774 log.cc:824] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:51.474802 24601 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c4ab2e4ddec246e494da465afe38958a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676
I20260621 07:09:51.479194 24774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: No bootstrap required, opened a new log
I20260621 07:09:51.486336 24774 raft_consensus.cc:348] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.486697 24774 raft_consensus.cc:729] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d263a4fd08e04fd0b4af494dcdf0921f, State: Initialized, Role: FOLLOWER
I20260621 07:09:51.487149 24774 consensus_queue.cc:260] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.488171 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.488556 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.489151 24774 sys_catalog.cc:564] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: configured and running, proceeding with master startup.
W20260621 07:09:51.498917 24793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 07:09:51.499136 24793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 07:09:51.496695 24707 master_runner.cc:186] Error getting master registration for 127.1.188.254:41741: OK, instance_id { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" instance_seqno: 1782025790378656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:09:51.508908 24707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:09:51.521916 24626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 4
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.254@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:09:51.537653 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.522688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36548)
0621 07:09:51.522916 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.522937 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.523066 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.523489 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.523495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.524028 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.524244 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.524257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.525814 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.525822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.527447 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.527463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.528477 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.528484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.528674 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.529365 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.529383 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.531777 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.534943 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.534951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.534956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.535486 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.535873 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.535879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.535883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.536022 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.536618 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.536623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.536819 (+ 196us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.536976 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.537263 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 07:09:51.537659 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.522765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.1.188.252:42387)
0621 07:09:51.523018 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.523024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.523048 (+ 24us) server_negotiation.cc:407] Connection header received
0621 07:09:51.523206 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.523211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.523299 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.523428 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.524397 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.525652 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.527592 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.528316 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.528440 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.531968 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.531973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.531976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.532007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.534785 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.535623 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.535634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.535636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.535757 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.536139 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.536144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.536148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.536478 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.536672 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.537088 (+ 416us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.537285 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260621 07:09:51.544392 24626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: Bootstrap starting.
I20260621 07:09:51.548983 24626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:51.550557 24626 log.cc:824] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:51.554332 24626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: No bootstrap required, opened a new log
I20260621 07:09:51.562820 24626 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.563293 24626 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ab2e4ddec246e494da465afe38958a, State: Initialized, Role: FOLLOWER
I20260621 07:09:51.563876 24626 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.564985 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.565431 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.566433 24626 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:51.566489 24601 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 07:09:51.566787 24601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:51.569708 24601 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.571154 24601 leader_election.cc:288] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [CANDIDATE]: Term 1 election: Requested vote from peers 006a6dc75e60484bade735b23468d12f (127.1.188.253:43877), d263a4fd08e04fd0b4af494dcdf0921f (127.1.188.252:42387)
I20260621 07:09:51.572458 24749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4ab2e4ddec246e494da465afe38958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d263a4fd08e04fd0b4af494dcdf0921f"
I20260621 07:09:51.572942 24749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:51.576077 24749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ab2e4ddec246e494da465afe38958a in term 1.
I20260621 07:09:51.576912 24577 leader_election.cc:302] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4ab2e4ddec246e494da465afe38958a, d263a4fd08e04fd0b4af494dcdf0921f; no voters:
I20260621 07:09:51.577450 24796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 07:09:51.587335 24796 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [term 1 LEADER]: Becoming Leader. State: Replica: c4ab2e4ddec246e494da465afe38958a, State: Running, Role: LEADER
I20260621 07:09:51.588340 24796 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.611676 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ab2e4ddec246e494da465afe38958a. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.612226 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:51.614051 24811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260621 07:09:51.617694 24580 tablet.cc:2369] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:51.633244 24700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } attempt: 5
W20260621 07:09:51.633765 24728 tablet.cc:2369] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 07:09:51.671926 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.601051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58832 (local address 127.1.188.253:43877)
0621 07:09:51.601283 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.601291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.609763 (+ 8472us) server_negotiation.cc:407] Connection header received
0621 07:09:51.618692 (+ 8929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.618699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.618806 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.618891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.629546 (+ 10655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.630992 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.642434 (+ 11442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.643473 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.643595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.656387 (+ 12792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.656395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.656401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.656446 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.660665 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.667651 (+ 6986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.667660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.667663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.667853 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.668943 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.668950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.668954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.669333 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.669583 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.671193 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.671502 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260621 07:09:51.673702 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.613078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.1.188.252:42387)
0621 07:09:51.613259 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.613267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.622197 (+ 8930us) server_negotiation.cc:407] Connection header received
0621 07:09:51.622659 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.622666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.622775 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.622861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.633128 (+ 10267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.635979 (+ 2851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.645067 (+ 9088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.645965 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.646088 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.661706 (+ 15618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.661714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.661732 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.661777 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.665980 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.668323 (+ 2343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.668329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.668335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.668495 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.670060 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.670066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.670070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.670419 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.670642 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.673082 (+ 2440us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.673323 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 21 07:09:51 dist-test-slave-hlq1 krb5kdc[24546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782025790, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.188.253@KRBTEST.COM for kudu/127.1.188.252@KRBTEST.COM
I20260621 07:09:51.685184 24811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 07:09:51.691073 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.603256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.1.188.254:41741)
0621 07:09:51.603495 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.603502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.610256 (+ 6754us) server_negotiation.cc:407] Connection header received
0621 07:09:51.610463 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.610472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.610602 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.610771 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.626226 (+ 15455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.627531 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.633645 (+ 6114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.634656 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.634823 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.656101 (+ 21278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.656110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.656116 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.656166 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.682447 (+ 26281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.686849 (+ 4402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.686857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.686862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.687062 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.687629 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.687639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.687646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.688162 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.688503 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.690082 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.690420 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:09:51.709456 24703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.634117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.188.252:42387 (local address 127.0.0.1:36568)
0621 07:09:51.634718 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0621 07:09:51.634745 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 07:09:51.634909 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 07:09:51.656509 (+ 21600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 07:09:51.656525 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 07:09:51.657353 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0621 07:09:51.657660 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.657680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.663224 (+ 5544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.663234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.668550 (+ 5316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 07:09:51.668574 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.671648 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.671658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 07:09:51.672071 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.672969 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 07:09:51.672997 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 07:09:51.677946 (+ 4949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 07:09:51.706074 (+ 28128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.706084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.706089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.706719 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.707306 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 07:09:51.707312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 07:09:51.707316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 07:09:51.707552 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 07:09:51.708341 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 07:09:51.708347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 07:09:51.708530 (+ 183us) client_negotiation.cc:769] Sending connection context
0621 07:09:51.708688 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 07:09:51.709048 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403}
I20260621 07:09:51.711047 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:51.635368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.1.188.252:42387)
0621 07:09:51.654810 (+ 19442us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.654820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.654967 (+ 147us) server_negotiation.cc:407] Connection header received
0621 07:09:51.655352 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.655360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.655795 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.658029 (+ 2234us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.658698 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.662879 (+ 4181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.669050 (+ 6171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:51.671265 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:51.694889 (+ 23624us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:51.695293 (+ 404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 07:09:51.695302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 07:09:51.695307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 07:09:51.695352 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 07:09:51.705881 (+ 10529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.706937 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.706943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.706946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.707149 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 07:09:51.707720 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 07:09:51.707725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 07:09:51.707729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 07:09:51.708176 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 07:09:51.710155 (+ 1979us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:51.710318 (+ 163us) server_negotiation.cc:299] Negotiation successful
0621 07:09:51.710562 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18557,"thread_start_us":17414,"threads_started":1}
I20260621 07:09:51.699069 24559 master_runner.cc:186] Error getting master registration for 127.1.188.253:43877: OK, instance_id { permanent_uuid: "006a6dc75e60484bade735b23468d12f" instance_seqno: 1782025790900394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 07:09:51.714408 24559 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 07:09:51.718251 24700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: Bootstrap starting.
I20260621 07:09:51.728302 24700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: Neither blocks nor log segments found. Creating new log.
I20260621 07:09:51.731863 24700 log.cc:824] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: Log is configured to *not* fsync() on all Append() calls
I20260621 07:09:51.741770 24749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [term 1 FOLLOWER]: Refusing update from remote peer c4ab2e4ddec246e494da465afe38958a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:51.742738 24796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:51.745489 24700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: No bootstrap required, opened a new log
I20260621 07:09:51.759620 24700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.760285 24700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006a6dc75e60484bade735b23468d12f, State: Initialized, Role: FOLLOWER
I20260621 07:09:51.760982 24700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } }
I20260621 07:09:51.767169 24824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.769879 24700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 07:09:51.770303 24674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [term 0 FOLLOWER]: Advancing to term 1
I20260621 07:09:51.773100 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ab2e4ddec246e494da465afe38958a. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.773428 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.775534 24820 mvcc.cc:205] Tried to move back new op lower bound from 7299177642951155712 to 7299177642354696192. Current Snapshot: MvccSnapshot[applied={T|T < 7299177642951155712}]
I20260621 07:09:51.776338 24674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [term 1 FOLLOWER]: Refusing update from remote peer c4ab2e4ddec246e494da465afe38958a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 07:09:51.770468 24675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4ab2e4ddec246e494da465afe38958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "006a6dc75e60484bade735b23468d12f"
I20260621 07:09:51.778240 24803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [LEADER]: Connected to new peer: Peer: permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 07:09:51.780830 24824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.782718 24675 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ab2e4ddec246e494da465afe38958a in term 1.
I20260621 07:09:51.794440 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.794729 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.794806 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.795110 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:51.799664 24811 catalog_manager.cc:1347] Generated new cluster ID: a41b921d2e8243b2be1f33004ac1be06
I20260621 07:09:51.799813 24811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 07:09:51.830292 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.830623 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a [sys.catalog]: This master's current role is: LEADER
I20260621 07:09:51.830504 24833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.830875 24833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.831157 24824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ab2e4ddec246e494da465afe38958a. Latest consensus state: current_term: 1 leader_uuid: "c4ab2e4ddec246e494da465afe38958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ab2e4ddec246e494da465afe38958a" member_type: VOTER last_known_addr { host: "127.1.188.254" port: 41741 } } peers { permanent_uuid: "006a6dc75e60484bade735b23468d12f" member_type: VOTER last_known_addr { host: "127.1.188.253" port: 43877 } } peers { permanent_uuid: "d263a4fd08e04fd0b4af494dcdf0921f" member_type: VOTER last_known_addr { host: "127.1.188.252" port: 42387 } } }
I20260621 07:09:51.831485 24824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f [sys.catalog]: This master's current role is: FOLLOWER
I20260621 07:09:51.840356 24811 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 07:09:51.843143 24811 catalog_manager.cc:1504] Loading token signing keys...
I20260621 07:09:51.888712 24811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ab2e4ddec246e494da465afe38958a: Generated new TSK 0
I20260621 07:09:51.890183 24811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 07:09:51.897099 24842 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260621 07:09:51.971091 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:51.963144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.1.188.253:43877)
0621 07:09:51.963368 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.963375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.967694 (+ 4319us) server_negotiation.cc:407] Connection header received
0621 07:09:51.969402 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.969409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.969512 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.969595 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.970820 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58842: BlockingRecv error: recv got EOF from 127.0.0.1:58842 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 07:09:51.973600 24779 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:51.963797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.1.188.254:41741)
0621 07:09:51.964004 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.964011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.969112 (+ 5101us) server_negotiation.cc:407] Connection header received
0621 07:09:51.971850 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.971857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.971966 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.972047 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.973325 (+ 1278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47696: BlockingRecv error: recv got EOF from 127.0.0.1:47696 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260621 07:09:51.976518 24816 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:51.967837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.1.188.252:42387)
0621 07:09:51.968056 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.968066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.974413 (+ 6347us) server_negotiation.cc:407] Connection header received
0621 07:09:51.975022 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.975029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.975145 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.975251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.976250 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36578: BlockingRecv error: recv got EOF from 127.0.0.1:36578 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260621 07:09:51.984508 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:51.982150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.1.188.253:43877)
0621 07:09:51.982359 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.982366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.982872 (+ 506us) server_negotiation.cc:407] Connection header received
0621 07:09:51.983135 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.983143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.983274 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.983388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.984248 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58846: BlockingRecv error: recv got EOF from 127.0.0.1:58846 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 07:09:51.993533 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:51.988818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.1.188.253:43877)
0621 07:09:51.989021 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:51.989027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:51.991960 (+ 2933us) server_negotiation.cc:407] Connection header received
0621 07:09:51.992177 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:51.992185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:51.992281 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:51.992400 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:51.993320 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58848: BlockingRecv error: recv got EOF from 127.0.0.1:58848 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 07:09:52.005628 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.003060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.1.188.253:43877)
0621 07:09:52.003262 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.003269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.004009 (+ 740us) server_negotiation.cc:407] Connection header received
0621 07:09:52.004219 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.004225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.004325 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.004449 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.005379 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58852: BlockingRecv error: recv got EOF from 127.0.0.1:58852 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 07:09:52.016006 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.013594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.1.188.253:43877)
0621 07:09:52.013845 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.013852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.014440 (+ 588us) server_negotiation.cc:407] Connection header received
0621 07:09:52.014672 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.014678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.014785 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.014892 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.015751 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58864: BlockingRecv error: recv got EOF from 127.0.0.1:58864 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260621 07:09:52.036150 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.033617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.1.188.253:43877)
0621 07:09:52.033821 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.033828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.034420 (+ 592us) server_negotiation.cc:407] Connection header received
0621 07:09:52.034641 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.034648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.034751 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.034892 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.035917 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58872: BlockingRecv error: recv got EOF from 127.0.0.1:58872 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 07:09:52.064138 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.061627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.1.188.253:43877)
0621 07:09:52.061852 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.061859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.062455 (+ 596us) server_negotiation.cc:407] Connection header received
0621 07:09:52.062682 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.062688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.062800 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.062915 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.063894 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58874: BlockingRecv error: recv got EOF from 127.0.0.1:58874 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260621 07:09:52.102667 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.100386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.1.188.253:43877)
0621 07:09:52.100602 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.100608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.100927 (+ 319us) server_negotiation.cc:407] Connection header received
0621 07:09:52.101130 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.101134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.101207 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.101294 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.102416 (+ 1122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58884: BlockingRecv error: recv got EOF from 127.0.0.1:58884 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260621 07:09:52.173197 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.170989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.1.188.253:43877)
0621 07:09:52.171250 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.171258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.171660 (+ 402us) server_negotiation.cc:407] Connection header received
0621 07:09:52.171874 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.171883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.171996 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.172139 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.172959 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58892: BlockingRecv error: recv got EOF from 127.0.0.1:58892 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260621 07:09:52.307063 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.305030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.1.188.253:43877)
0621 07:09:52.305320 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.305330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.305583 (+ 253us) server_negotiation.cc:407] Connection header received
0621 07:09:52.305817 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.305821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.305895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.306015 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.306790 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58898: BlockingRecv error: recv got EOF from 127.0.0.1:58898 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 07:09:52.501870 24793 catalog_manager.cc:1259] Loaded cluster ID: a41b921d2e8243b2be1f33004ac1be06
I20260621 07:09:52.502022 24793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: loading cluster ID for follower catalog manager: success
I20260621 07:09:52.504755 24793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: acquiring CA information for follower catalog manager: success
I20260621 07:09:52.506615 24793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d263a4fd08e04fd0b4af494dcdf0921f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 07:09:52.568535 24705 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 07:09:52.566388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.1.188.253:43877)
0621 07:09:52.566630 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:52.566639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:52.566855 (+ 216us) server_negotiation.cc:407] Connection header received
0621 07:09:52.567021 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:52.567025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:52.567103 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:52.567193 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 07:09:52.568340 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45988: BlockingRecv error: recv got EOF from 127.0.0.1:45988 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 07:09:52.902653 24842 catalog_manager.cc:1259] Loaded cluster ID: a41b921d2e8243b2be1f33004ac1be06
I20260621 07:09:52.902791 24842 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: loading cluster ID for follower catalog manager: success
I20260621 07:09:52.905807 24842 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: acquiring CA information for follower catalog manager: success
I20260621 07:09:52.908215 24842 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 006a6dc75e60484bade735b23468d12f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 07:09:53.093367 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:53.085713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.1.188.253:43877)
0621 07:09:53.086745 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:53.086750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:53.086771 (+ 21us) server_negotiation.cc:407] Connection header received
0621 07:09:53.086854 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:53.086858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:53.087044 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:53.087170 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:53.087728 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.088594 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.090535 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.091088 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.091198 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:53.091913 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:53.092642 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:53.092772 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:53.092903 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 07:09:53.092987 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":740,"threads_started":1}
I20260621 07:09:53.109737 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:53.100538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.1.188.252:42387)
0621 07:09:53.103803 (+ 3265us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:53.103808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:53.103827 (+ 19us) server_negotiation.cc:407] Connection header received
0621 07:09:53.103945 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:53.103950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:53.104130 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:53.104330 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:53.104796 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.105571 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.106984 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.107644 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.107797 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:53.108364 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:53.109042 (+ 678us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:53.109235 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:53.109288 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 07:09:53.109387 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3140,"thread_start_us":3026,"threads_started":1}
I20260621 07:09:53.127847 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:53.116830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.1.188.253:43877)
0621 07:09:53.117066 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:53.117077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:53.119374 (+ 2297us) server_negotiation.cc:407] Connection header received
0621 07:09:53.119993 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:53.120004 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:53.120147 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:53.120240 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:53.121258 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.122931 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.125333 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.126051 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.126147 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:53.126831 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:53.126980 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:53.127049 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:53.127311 (+ 262us) server_negotiation.cc:299] Negotiation successful
0621 07:09:53.127451 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 07:09:53.132117 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:53.116383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.1.188.254:41741)
0621 07:09:53.118895 (+ 2512us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:53.118902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:53.124051 (+ 5149us) server_negotiation.cc:407] Connection header received
0621 07:09:53.124608 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:53.124616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:53.124865 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:53.125042 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:53.125850 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.126993 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.129143 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.129846 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.129947 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:53.130742 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:53.131319 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:53.131428 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:53.131673 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 07:09:53.131776 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2339,"thread_start_us":2224,"threads_started":1}
I20260621 07:09:53.135344 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 07:09:53.119593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.1.188.252:42387)
0621 07:09:53.119831 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 07:09:53.119840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 07:09:53.128596 (+ 8756us) server_negotiation.cc:407] Connection header received
0621 07:09:53.128875 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 07:09:53.128883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 07:09:53.128989 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 07:09:53.129063 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 07:09:53.129650 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.130788 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.132890 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 07:09:53.133512 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 07:09:53.133631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 07:09:53.134385 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 07:09:53.134581 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 07:09:53.134689 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 07:09:53.134868 (+ 179us) server_negotiation.cc:299] Negotiation successful
0621 07:09:53.134961 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 07:09:53.139376 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24559
I20260621 07:09:53.140656 24621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:53.269953 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24559
W20260621 07:09:53.296013 24649 connection.cc:535] server connection from 127.0.0.1:58798 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260621 07:09:53.296800 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24633
I20260621 07:09:53.298297 24695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:53.423856 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24633
I20260621 07:09:53.446997 1779 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRiL99G/build/asan/bin/kudu pid 24707
I20260621 07:09:53.448362 24769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 07:09:53.570684 1779 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRiL99G/build/asan/bin/kudu with pid 24707
2026-06-21T07:09:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5083 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9984 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374812 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST